{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——LightGBM\n",
    "原始特征+tfidf特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，对LightGBM的超参数进行调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 读取数据\n",
    "# 这里使用原始特征+tf_idf特征，log(x+1)特征为原始特的单调变换，加上log特征对决策树模型影响不大\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "\n",
    "train1 = pd.read_csv(dpath +\"Otto_FE_train_org.csv\")\n",
    "train2 = pd.read_csv(dpath +\"Otto_FE_train_tfidf.csv\")\n",
    "\n",
    "#去掉多余的id\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字，LightGBM不支持字符串格式的特征输入/标签输入\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM超参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM的主要的超参包括：\n",
    "1. 树的数目n_estimators 和 学习率 learning_rate\n",
    "2. 树的最大深度max_depth 和 树的最大叶子节点数目num_leaves（注意：XGBoost只有max_depth，LightGBM采用叶子优先的方式生成树，num_leaves很重要，设置成比 2^max_depth 小）\n",
    "3. 叶子结点的最小样本数:min_data_in_leaf(min_data, min_child_samples)\n",
    "4. 每棵树的列采样比例：feature_fraction/colsample_bytree\n",
    "5. 每棵树的行采样比例：bagging_fraction （需同时设置bagging_freq=1）/subsample\n",
    "6. 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)\n",
    "\n",
    "7. 两个非模型复杂度参数，但会影响模型速度和精度。可根据特征取值范围和样本数目修改这两个参数\n",
    "1）特征的最大bin数目max_bin：默认255；\n",
    "2）用来建立直方图的样本数目subsample_for_bin：默认200000。\n",
    "\n",
    "对n_estimators，用LightGBM内嵌的cv函数调优，因为同XGBoost一样，LightGBM学习的过程内嵌了cv，速度极快。\n",
    "其他参数用GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相同的交叉验证分组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用lightgbm内嵌的交叉验证(cv)，可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证，且速度相对较慢\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 394\n",
      "best cv score: 0.4767505203274461\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  5.2min remaining:  2.6min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  7.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=31,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4, param_grid={'num_leaves': range(50, 90, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10) #50,60,70,80\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = 3, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47751951846390295\n",
      "{'num_leaves': 50}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEHCAYAAABm9dtzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9dn//9c7O4SwJqCsYVURIWhkUYu4YLWtqK0ICO4KWpe6tP3a3927i73bu7V1rVTBfRdsbaXeVsUFV0CCIosKBAgQEAj7Tkhy/f6YE4yYhBBmcjLJ9Xw85pE561wfhuQ9n885c47MDOeccy4aEsIuwDnnXMPhoeKccy5qPFScc85FjYeKc865qPFQcc45FzVJYRcQpszMTMvOzg67DOeciytz5szZYGZZlS1r1KGSnZ1NXl5e2GU451xckbSiqmUxHf6SdLakRZLyJd1ezXoXSjJJucH0GElzKzzKJOUEy0ZKmidpoaQ7K+wjVdLk4LVmScqOZducc859W8xCRVIiMAE4B+gNjJbUu5L1MoCbgFnl88zsWTPLMbMc4BKgwMzmSmoD/Bk4w8yOBdpJOiPY7Cpgs5n1AO4B/hSrtjnnnKtcLHsqA4B8M1tmZsXAC8B5laz3O+BOYE8V+xkNPB887wYsNrOiYPpN4EfB8/OAJ4PnfwfOkKTDa4JzzrlDEctQ6QCsqjBdGMzbT1J/oJOZvVLNfkbydajkA0dLypaUBJwPdDrw9cysBNgKtDncRjjnnKu5WIZKZb2E/Rcak5RAZJjqtip3IA0EdpnZAgAz2wxcB0wG3gcKgJKavF6FfY6TlCcpr6ioqJJNnHPO1VYsQ6WQr3sRAB2BNRWmM4A+wHRJBcAgYGr5wfrAKL7upQBgZv82s4FmNhhYBCw58PWCXkwLYNOBRZnZJDPLNbPcrKxKz4hzzjlXS7EMldlAT0ldJaUQCYip5QvNbKuZZZpZtpllAzOB4WaWB/t7MiOIHIvZT1Lb4Gcr4MfAI8GiqcBlwfMLgbfNL8HsnHN1KmbfUzGzEkk3AK8DicBjZrZQ0h1AnplNrX4PDAEKzWzZAfPvk9QveH6HmS0Onj8KPC0pn0gPZVR0WuLixaxlG+mW1YysjNSwS3Gu0VJj/jCfm5tr/uXHhmHhmq18//4P6NCyCc9dM5AubdLDLsm5BkvSHDPLrWyZX/vLNQj3TFtCRloSu4pLGPHQDBav2x52Sc41Sh4qLu7NXbWFN79Yx/gh3Zg8fjAAIyfOYH7h1pArc67x8VBxce/uaYtpnZ7C5Sd3pVe7DF68djBNU5K4+OGZzC741gmAzrkY8lBxcW12wSbeW1zEtad2o1lq5LyTLm3S+ft1g8lqnsolj87i/SX+fSTn6oqHiotrd72xiKyMVC4ZlP2N+Ue2aMKU8YPpmtmMq57I47UFa8Mp0LlGxkPFxa2P8jcwc9kmrh/anSYpid9antkslReuGUTv9s25/rlP+OenhSFU6Vzj4qHi4pKZ8Zc3FnFkizRGDehc5XotmibzzNUDGZDdmlunfMYzM6u8DYRzLgo8VFxcmr64iE9WbuHG03uSlvztXkpFzVKTePyKEzntqLb88l8LmPju0jqq0rnGx0PFxR0z4+43FtOpdRNG5Has0TZpyYk8NPYEvt/3SP73P19y9xuLaMxf/HUuVhr17YRdfHrj83XMX72VP1/Yl+TEmn8uSklK4P5R/UlPSeT+t/PZsbeU//7BMfhtd5yLHg8VF1fKyox7pi2mW2Y6F/TvcPANDpCYIP74w740TUnisQ+Xs6u4hN9fcByJCR4szkWDh4qLK/83/yu+XLud+0blkHQIvZSKEhLEr8/tTbPUJB54J5+dxaXcfVG/Q+r1OOcq56Hi4kZpmXHvm4s5ql0G5/Ztf1j7ksRPv3sU6alJ/Om1L9ldXMIDFx9/0IP+zrnq+UczFzdenruapUU7uWVYTxKiNFx13dDu/O68Y3nzi/Vc9eRsdu4tOfhGzrkqeai4uLCvtIx731zCse2b891jj4jqvi8ZnM1dI/oxY+lGLnl0Flt374vq/p1rTDxUXFz4x5xCVm7axW1n9YrJ2Vo/OqEjEy4+nvmrtzJ60kw27tgb9ddwrjHwUHH13t6SUu5/awk5nVpy2lFtY/Y65xx3JA9fmsvSoh1cNHEGa7fuidlrOddQeai4em/y7FWs2bqHn551VMy/UzL0qLY8deUA1m3by4iJH7Fy466Yvp5zDY2HiqvX9uwr5YG38xnQtTUn92hTJ685sFsbnr16INt2lzBi4kfkr/e7SDpXUzENFUlnS1okKV/S7dWsd6Ekk5QbTI+RNLfCo0xSTrBstKT5kuZJek1SZjA/R9LMYP08SQNi2TZXN56ZuYL12/dy27DYHEupSr9OLZk8fhClZXDRxJksWO13kXSuJmIWKpISgQnAOUBvYLSk3pWslwHcBMwqn2dmz5pZjpnlAJcABWY2V1IScB9wmpn1BeYBNwSb3Qn8NtjmV8G0i2M795bw4PSlfKdnJgO71U0vpaKjj2jOi9cOJi0pgdEPz2TOCr+LpHMHE8ueygAg38yWmVkx8AJwXiXr/Y5IAFR1VHQ08HzwXMEjXZGPrc2BNcEyC6YBWlSY7+LUEx8VsHFnMbcO6xVaDV0z03nxupNok57C2Ec+5oMlG0Krxbl4EMtQ6QCsqjBdGMzbT1J/oJOZvVLNfkYShIqZ7QOuA+YTCY3ewKPBejcDf5a0CvgL8IvKdiZpXDA8lldU5LeZra+27dnHpPeWccbRbenfuVWotXRo2YQp1w6mc+umXPnEbKZ9vi7Uepyrz2IZKpUNgO+/1rikBOAe4LYqdyANBHaZ2YJgOplIqPQH2hMZ/ioPj+uAW8ysE3ALX4fNNwswm2RmuWaWm5WVdciNcnXjsQ+Ws3X3Pm4JsZdSUduMNCaPH8QxR2Zw7TNzeHnu6rBLcq5eimWoFAKdKkx35JtDUhlAH2C6pAJgEDC1/GB9YBRfD30B5ACY2VKL3AxjCnBSsOwy4KXg+YtEht9cHNqyq5hH31/O2cceQZ8OLcIuZ7+WTVN45uqBnNClFTdPnssLH68MuyTn6p1YhspsoKekrpJSiATE1PKFZrbVzDLNLNvMsoGZwHAzy4P9PZkRRI7FlFsN9JZU3sUYBnwRPF8DnBo8Px1YEptmuVib9N4ydhSX1JteSkUZack8ecUAhvTM4vaX5vPI+8vCLsm5eiVmVyk2sxJJNwCvA4nAY2a2UNIdQJ6ZTa1+DwwBCs1s/2+tma2R9FvgPUn7gBXA5cHia4D7gjPE9gDjotsiVxc27NjL4x8WcG7f9hx1REbY5VSqSUoiky49gZtfmMv//N8X7Nxbyk1n9PCbfTkHqDHfUjU3N9fy8vLCLsNV8D+vfM5jHy7nzVtPpVtWs7DLqVZJaRk//8c8XvpkNeOGdOMX5xztweIaBUlzzCy3smV+PxVXb6zbtoenZ67gh8d3rPeBApCUmMBfLuxHekpSZMhubwn/c16fqF2W37l45KHi6o0J7+RTWmb85IyeYZdSYwkJ4o7zjiU9NYmH3l3Krr0l/GVEv1rfldK5eOeh4uqFws27eP7jlVx0Yic6tW4adjmHRBK3n3M0GWlJ/Pn1RewqLuWvF/cnNcnvIukaH/845eqFB97OR4gbTusRdim1dv1pPfj1ub154/N1XP1kHruK/S6SrvHxUHGhW7FxJy/OKeTigZ1p37JJ2OUclitO7sqdP+rLh/kbuOyxj9m2x+8i6RoXDxUXuvveWkJyovjx0O5hlxIVF53YiftH9+fTlVsY8/AsNu0sDrsk5+qMh4oLVf76Hfzr09VcOjibts3Twi4nan7Qtz2TLj2BReu2M3LiDNZv87tIusbBQ8WF6t43F9MkOZHxQ7qFXUrUnX50O5644kRWb9nNiIkzWLXJ7yLpGj4PFReaL77axivzvuKKk7vSpllq2OXExEndM3nm6oFs3lnMRRNnsLRoR9glORdTHiouNPdMW0xGWhLXfKfh9VIqOr5zK14YN5jikjJGTpzB52u2hV2SczHjoeJCMb9wK298vo5rvtONFk2Twy4n5nq3b86UaweTnJjAqEkz+GTl5rBLci4mPFRcKO6atoiWTZO54uTssEupM92zmjFl/GBaNk1h7COz+Gip30XSNTweKq7OzVmxmemLirj21O5kpDX8XkpFnVo35cVrB9OhZROueHw2b3/pd5F0DYuHiqtzd09bRGazFC4d3CXsUkLRrnkak8cPpme7Zox7ag7/N++rsEtyLmo8VFydmrF0Ix/mb+S6oT1omtJ4Lz3XOj2F564ZRE6nltz4/CdMyVsVdknORYWHiqszZsbd0xZxRPM0xgzsHHY5oWuelsxTVw3g5B6Z/Pzv83jiw+Vhl+TcYfNQcXXmvSUbmF2wmetP70Fasl/BF6BpShKPXJbLWb3b8Zt/f86Ed/LDLsm5w+Kh4uqEmXH3G4vo0LIJI3M7hV1OvZKalMiEMcdzfk57/vz6Iv702pc05juyuvgW01CRdLakRZLyJd1ezXoXSjJJucH0GElzKzzKJOUEy0ZLmi9pnqTXJGVW2M+NwestlHRnLNvmDs1bX6zns8Kt/OSMnqQk+WeZAyUnJnD3RTlcPLAzD05fyq+nLqSszIPFxZ+YHSmVlAhMAIYBhcBsSVPN7PMD1ssAbgJmlc8zs2eBZ4PlxwEvm9lcSUnAfUBvM9sQBMcNwG8knQacB/Q1s72S2saqbe7QlJUZd01bTHabpvzw+A5hl1NvJSSI35/fh2apkdsT79xbyp9+dJzfRdLFlVj+bx0A5JvZMjMrBl4g8kf/QL8D7gSquozraOD54LmCR7okAc2BNcGy64A/mtleADNbH5VWuMP22sK1fPHVNm4+s5f/gTwISfzinKO5dVgv/vFJITc+/ynFJWVhl+VcjcXyN7wDUPE8ycJg3n6S+gOdzOyVavYzkiBUzGwfkfCYTyRMegOPBuv1Ar4jaZakdyWdWNnOJI2TlCcpr6ioqBbNcoeitMy4e9pierZtxrn92oddTlyQxE1n9OS/f9Cb/yxYyzVP5bG7uDTsspyrkViGiiqZt3+QWFICcA9wW5U7kAYCu8xsQTCdTCRU+gPtgXnAL4LVk4BWwCDgZ8CUoDfzzQLMJplZrpnlZmVl1aZd7hD8+7M15K/fwc1n9iIxobL/Eq4qV53SlT/+8DjeW1LEZY9/zHa/i6SLA7EMlUKg4mk+Hfl6qAogA+gDTJdUQCQMppYfrA+M4uuhL4AcADNbapHTY6YAJ1V4vZcs4mOgDMjEhaaktIx731zMMUc255w+R4RdTlwaNaAz943qzycrNjP2kVls2eV3kXT1WyxDZTbQU1JXSSlEAmJq+UIz22pmmWaWbWbZwExguJnlwf6ezAgix2LKrQZ6SyrvYgwDvgie/ws4Pdi2F5AC+BX7QvTSJ6sp2LiLW4f1IsF7KbU2vF97Hhp7Al+s3c7IiTNZv93vIunqr5iFipmVEDkz63Uif/inmNlCSXdIGl6DXQwBCs1sWYV9rgF+C7wnaR6RnssfgsWPAd0kLSASRJeZn+wfmuKSMu57awn9OrbgzGP8RLzDdWbvdjx++Yms3LSLkRNnsnrL7rBLcq5Sasx/d3Nzcy0vLy/sMhqkZ2au4Jf/WsCTVw7g1F5+7Cpa5qzYxOWPz6Z5WjLPXD2QrpnpYZfkGiFJc8wst7Jlfn6ni7o9+0p54O18cru0YkhPP6wVTSd0ac3z1wxi975SRjw0gy/X+l0kXf3ioeKi7rlZK1m7bQ+3nXUUlZyA5w5Tnw4tmDJ+EIkJMGrSTD5btSXskpzbz0PFRdWu4hL+Nj2fk7q3YXD3NmGX02D1aJvBi+NPIiMtiTGPzGLWso1hl+Qc4KHiouypGSvYsKOY287qFXYpDV7nNk15cfxJtGueymWPf8z0RX4RCRc+DxUXNdv37GPiu0sZelQWJ3RpHXY5jcIRLdKYMn4w3TKbcc1Tefxnvt9F0oXLQ8VFzeMfFrB51z5uHea9lLrUplkqz48bxHEdWnD9c5/wjzmFYZfkGjEPFRcVW3ft4+H3l3FW73b07dgy7HIanRZNknn6qoEM7t6G2178jKdnFIRdkmukPFRcVDz8/jK27ynhFu+lhCY9NYlHLzuRM49py3+/vJAHpy8NuyTXCHmouMO2aWcxj3+4nB/0PZJjjmwedjmNWlpyIg+OPYFz+7XnT699yZ9f97tIuroVs5t0ucZj4rtL2b2vlJvP9F5KfZCcmMC9I3NIT0lkwjtL2bm3lF/9oLdff83VCQ8Vd1jWb9/DkzMKOD+nAz3aNgu7HBdITBD/+8PjIkNiHyxn594S/vijvn77ARdzHirusPztnaXsKzV+cmbPsEtxB5DEL79/DOmpSdz/1hJ2FZdyz8gcUpJ81NvFjoeKq7U1W3bz3KyVjDihI13a+IUN6yNJ3DqsF81SE/nDq1+ye18pfxtzPGnJiWGX5hoo/8jiau2Bd/IBuPEM76XUd+OGdOf3F/ThnUXrueLx2ezYWxJ2Sa6B8lBxtbJq0y6mzF7FqAGd6NCySdjluBoYM7ALd1/Uj48LNjH2kVls3eW3J3bR56HiauW+t5aQmCCuP61H2KW4Q3BB/45MuPh4Pl+zjZGTZlC0fW/YJbkGxkPFHbJlRTt46ZNCLhnUhXbN08Iuxx2is/scwSOX5VKwcScjJ85gjd9F0kWRh4o7ZPe+uYS05ESuHdo97FJcLQ3plcXTVw2kaPteRjw0gxUbd4ZdkmsgYhoqks6WtEhSvqTbq1nvQkkmKTeYHiNpboVHmaScYNloSfMlzZP0mqTMA/b102BffsvBGFi0djv/nreGy07KJrNZatjluMNwYnZrnrtmELuKSxjx0AwWr9sedkmuAYhZqEhKBCYA5wC9gdGSeleyXgZwEzCrfJ6ZPWtmOWaWA1wCFJjZXElJwH3AaWbWF5gH3FBhX52AYcDKWLWrsbv3zcU0S0li/JBuYZfiouC4ji2YPH4wBoycOIP5hVvDLsnFuVj2VAYA+Wa2zMyKgReA8ypZ73fAncCeKvYzGng+eK7gka7IfWqbA2sqrHsP8HPAL3YUAwtWb+U/C9Zy5Sldadk0JexyXJT0apfBi+MH0zQliYsfnsnsgk1hl+TiWCxDpQOwqsJ0YTBvP0n9gU5m9ko1+xlJECpmtg+4DphPJEx6A48G+xoOrDazz6LVAPdN90xbTIsmyVz1na5hl+KiLDsznRevHUxWRiqXPDqL95cUhV2Si1OxDJXKLjK0vwchKYFIz+K2KncgDQR2mdmCYDqZSKj0B9oTGf76haSmwH8BvzpoUdI4SXmS8oqK/Benpj5duZm3vlzPuCHdaJ6WHHY5Lgbat2zC5PGDyW6TzlVP5PH6wrVhl+Ti0EFDRVJ3SanB86GSbpJUk7swFQKdKkx35JtDVRlAH2C6pAJgEDC1/GB9YBRfD30B5ACY2VKLXM97CnAS0B3oCnwW7Ksj8ImkIw4syswmmVmumeVmZWXVoBkO4O5pi2mTnsLlJ2WHXYqLoayMVF4YN4je7Zvz42c/4V+frg67JBdnatJT+QdQKqkHkaGmrsBzNdhuNtBTUldJKUQCYmr5QjPbamaZZpZtZtnATGC4meXB/p7MCCLHYsqtBnpLKk+DYcAXZjbfzNpW2FchcLyZ+UetKJi1bCPvL9nAdUO7k57ql4tr6Fo2TeGZqwdyYnYrbpkyl2dnrQi7JBdHahIqZWZWAlwA3GtmtwBHHmyjYJsbgNeBL4ApZrZQ0h3B8Y+DGQIUmtmyCvtcA/wWeE/SPCI9lz/UYF+ulsyMu6Ytpm1GKmMHdQm7HFdHmqUm8cQVAxjaK4v/+ucCJr3nd5F0NVOTj537JI0GLgPODebVaFDdzF4FXj1gXqXHPcxs6AHT04kMiR243kPAQwd53eya1OcO7sP8jXy8fBN3nHesX9m2kUlLTmTiJbncMnkuf3j1S3bsLeWWM3sSOfHSucrVJFSuAK4Ffm9myyV1BZ6JbVmuPoj0UhbRvkUaI0/sdPANXIOTkpTA/aP70zQlkfvfWsLOvSX88vvHeLC4Kh00VMzscyJfTkRSKyDDzP4Y68Jc+N5ZtJ5PV27hf394HKlJ3ktprBITxJ9+1Pcbd5H8/QXH+V0kXaUOGiqSpgPDg3XnAkWS3jWzW2NcmwuRmXHXG4vp3LopF57QMexyXMgSEsSvz+1Nemrkvve7iku566J+JCf65QPdN9Vk+KuFmW2TdDXwuJn9OjhI7hqw1xeuZeGabdw1wv9wuAhJ/Oy7R5OemsSdry1iV3EpD1zc34+11UNmxrY9JRRt3xt57Nj79fNg+oqTsjnt6LZRf+2ahEqSpCOBi4h8wdA1cGVlxj3TltA9K53z+3c4+AauUfnx0B40S03iVy8v5KonZzPpklw/1byO7NlXWmVIHDhdXFL2re2TE0VWs1SyMlLZW1Iakxpr8j/hDiKnBX9oZrMldQOWxKQaVy+8Mv8rFq3bzl9H9/dxc1epSwdn0zQliZ///TMufexjHrv8RFo08Sst1EZpmbFx58FDomj7Xrbv+fZtoCVo3TSFrIxIWHTLTN//PCsjdX+IZGWk0qJJcsxPslDki+mNU25uruXl5YVdRr1SUlrGWfe8R0pSAq/e9B0SPFRcNV6d/xU/eeFTerXL4KkrB9DGb4cABMNPu0so2rGH9RXCYcOO4m+Fxqadeymr5M9wRmoSWRmpZFYSDhWft05PqfMhaklzzCy3smU1OVDfEfgrcDKRa3d9APzEzAqjWqWrF/41dw3LNuxk4iUneKC4g/recUfSJCWRa5+ew8hJM3nmqoEc0aLh3g20fPhpfTU9ig3B8+LSbw8/pSQm7A+KDi2bkNOp5beCom1GKpnNUmmSEp/Hqg7aU5E0jchlWZ4OZo0FxpjZsBjXFnPeU/mmfaVlnH7XdFo2SWHqDSf7dxFcjc1ctpGrnphN62YpPHf1IDq1bhp2STVWUlrGpp3FkaCoYtipPCi27618+KlNegqZFXsSB/Qm2makktUsjeZNkhrE79Vh9VSALDN7vML0E5Jujk5prj55Ma+QVZt2c8cVfRrEf3xXdwZ1a8Oz1wzissc+5sKHPuLZqwfSo21GaPVUNfx0YGhs2LGXjTuLqeyzdcXhp2PaN2dIJaHRNhh+SvIzJPerSahskDSWr68WPBrYGLuSXBj27Cvlr28v4fjOLRnay6/e7A5dTqeWTB4/iLGPfMxFE2fy1JUD6NOhRVRfY3dx+dlPe6o9sL1hR/FBh586tmpK/86tGtzwU9hqEipXAg8QufeJAR8RuXSLa0Be+HglX23dw19G9PNeiqu1o49ozpTxgxj7yCxGPzyTJ64YwAldWlW7TUlpGRt3Ftfo7KcdVQ4/RQIhs1kK3ds2a/DDT/VZrc7+knSzmd0bg3rqlB9TidhdXMqQP79D96x0nr9mkP/SucNWuHkXYx+Zxfrte/nt8GNJSUqoMjQ27api+Ckt6dtnPVUy3bqpDz/VtcM9plKZW4G4DxUX8fTMAoq27+VvY473QHFR0bFVU6ZcO5hLHvmYn/396wtwpCQl7A+FTq2bcnyXVlWGhn9TPz7VNlT8L08DsWNvCQ+9u4zv9MzkxOzWYZfjGpC2GWn88/qTWLB6G63TI1/Oa57mw08NXW1DpfF+Y7KBefKjAjbtLOa2s44KuxTXADVNSWJAV/+w0phUGSqStlN5eAhoErOKXJ3ZunsfE99dypnHtCWnU8uwy3HONQBVhoqZhXeSuasTj36wnG17SrhlWK+wS3HONRB+ykQjtXlnMY99sJzvHXcEx7aP7ncJnHONV0xDRdLZkhZJypd0ezXrXSjJJOUG02Mkza3wKJOUEywbLWm+pHmSXpOUGcz/s6Qvg/n/lOTjOdWY+N4ydhaXcPOZ3ktxzkVPzEJFUiIwATgH6A2MltS7kvUyiNyueFb5PDN71sxyzCwHuAQoMLO5kpKA+4DTzKwvMA+4IdhsGtAnmL8Y+EWs2hbvirbv5cmPCjivX3t6tfNRTudc9MSypzIAyDezZWZWDLwAnFfJer8D7gT2VLGf0Xx9iRgFj3RFzktsDqwBMLM3zKz867YzAb8HbhUenL6U4tIyfuK9FOdclB00VCRtl7TtgMeqYIipWzWbdgBWVZguDOZV3Hd/oJOZvVLNfkYShIqZ7QOuA+YTCZPewKOVbHMl8J8q2jNOUp6kvKKiompetmFau3UPz8xawY+O70DXzPSwy3HONTA16ancDfyMSCB0BH4KPEyk5/FYNdtV9g2n/acoS0ogcj2x26rcgTQQ2GVmC4LpZCKh0h9oT2T46xcHbPNfQAnwbGX7NLNJZpZrZrlZWY3vwokT3snHzLjx9J5hl+Kca4BqEipnm9lEM9tuZtvMbBLwPTObDFR3pbhCoFOF6Y4EQ1WBDKAPMF1SATAImFp+sD4wiq+HvgByAMxsqUUuWjYFOKl8oaTLgB8Qud+Lf0HzAIWbd/HC7JVclNspru534ZyLHzUJlTJJF0lKCB4XVVhW3R/u2UBPSV0lpRAJiKn7NzTbamaZZpZtZtlEjoMMN7M82N+TGUGkR1RuNdBbUnkXYxjwRbD+2cD/C/axqwbtanT++lY+krjh9B5hl+Kca6BqEipjiJyBtT54XAKMldSEr8+8+pbgoPkNwOtE/vBPMbOFku6QNLwGrzsEKDSzZRX2uQb4LfCepHlEei5/CBY/QKT3My04DfmhGrxGo1GwYSd//6SQMQM7c2QLvyCCcy42anXp+4aiMV36/pbJc3ltwVre/flQ2mY03HuIO+dir7pL39fk7K+OwZle6yWtk/QPSX66bhxZsm47/5q7mktP6uKB4pyLqZoMfz1O5FhIeyJngP07mOfixL1vLqFpciLjh3QPuxTnXANXk1DJMrPHzawkeDwBNL5zcePU52u28X/zv+KqU7rSOj0l7HKccw1cTUJlg6SxkhKDx1hgY6wLc9Fx97TFNE9L4qrvVPc9Veeci46ahMqVwEXAWuAr4ELgilgW5aLjs1VbePOLdYwb0o0WTZLDLsc51wgcNFTMbKWZDTezLDNra2bnAz+sg+WHWxAAABNnSURBVNrcYbp72mJaNU3m8pO7hl2Kc66RqO0FJW+NahUu6vIKNvHu4iKuPbU7zVJre9do55w7NLUNlcqu6+XqkbveWExms1QuHZwddinOuUaktqHSeL8xGQc+yt/AjGUbuf607jRJSQy7HOdcI1LluIik7VQeHgL8Oh/1lJlx17TFHNkijdEDOoddjnOukakyVMzMbwkYh95dXMScFZv5/QV9SEv2Xopzrm7F9B71rm6ZGXdPW0zHVk0YcUKng2/gnHNR5qHSgEz7fB3zCrfykzN6kpLkb61zru75X54Goqws0kvplpnOBf07HHwD55yLAQ+VBuLVBV/x5drt/OTMniQl+tvqnAuH//VpAErLjHumLaZXu2b8oG/7sMtxzjViHioNwMtzV7O0aCe3nNmLxAT/XqpzLjweKnFuX2kZ9721hGPbN+e7xx4RdjnOuUYupqEi6WxJiyTlS7q9mvUulGSScoPpMcF95ssfZZJygmWjJc2XNE/Sa5Iyg/mtJU2TtCT42SqWbasvXvqkkBUbd3HrsF4keC/FOReymIWKpERgAnAO0BsYLal3JetlADcBs8rnmdmzZpZjZjnAJUCBmc2VlATcB5xmZn2BecANwWa3A2+ZWU/grWC6QdtbUsr9b+WT06klpx/dNuxynHMupj2VAUC+mS0zs2LgBeC8Stb7HXAnsKeK/YwGng+eK3ikSxLQHFgTLDsPeDJ4/iRw/mG3oJ6bMnsVq7fs5razehH553DOuXDFMlQ6AKsqTBcG8/aT1B/oZGavVLOfkQShYmb7gOuA+UTCpDfwaLBeOzP7KljvK6DSj+6SxknKk5RXVFR0yI2qL/bsK+Wvb+czILs1p/TIDLsc55wDYhsqlX103n+BSkkJwD3AbVXuQBoI7DKzBcF0MpFQ6Q+0JzL89YtDKcrMJplZrpnlZmVlHcqm9cozM1ewfvte76U45+qVWIZKIVDxAlQd+XqoCiAD6ANMl1QADAKmlh+sD4zi66EvgBwAM1tqZgZMAU4Klq2TdCRA8HN99JpSv+zcW8JD7y7llB6ZDOzWJuxynHNuv1iGymygp6SuklKIBMTU8oVmttXMMs0s28yygZnAcDPLg/09mRFEjsWUWw30llTexRgGfBE8nwpcFjy/DHg5Ns0K35MzCtiwo5hbz+oVdinOOfcNMbvPrJmVSLoBeB1IBB4zs4WS7gDyzGxq9XtgCFBoZssq7HONpN8C70naB6wALg8W/xGYIukqYCWRQGpwtu3Zx8R3l3H60W05vnOjOGvaORdHFBlFapxyc3MtLy8v7DIOyb1vLubeN5fwyo2n0KdDi7DLcc41QpLmmFluZcv8G/VxZMuuYh59fznfPbadB4pzrl7yUIkjD7+/jB3FJdwyzI+lOOfqJw+VOLFxx14e/7CAH/Rtz9FHNA+7HOecq5SHSpx46N2l7NlXys1n9gy7FOecq5KHShxYv20PT81YwQX9O9I9q1nY5TjnXJU8VOLAhHfyKS0zfnKG91Kcc/Wbh0o9t3rLbp7/eBUjcjvRuU3TsMtxzrlqeajUcw+8vQSAG0/vEXIlzjl3cB4q9djKjbt4Ma+Qiwd2pn3LJmGX45xzB+WhUo/d99YSEhPEj4d2D7sU55yrEQ+Veip//Q7++Wkhlw7uQtvmaWGX45xzNeKhUk/d99YS0pITufZU76U45+KHh0o99OXabfz7szVccXI2bZqlhl2Oc87VmIdKPXTPtMVkpCUx7jveS3HOxRcPlXpmfuFWXl+4jqtP6UaLpslhl+Occ4fEQ6WeuXvaIlo2TebKU7LDLsU55w6Zh0o9MmfFZt5ZVMT4Id3JSPNeinMu/nio1CN3T1tEZrMULjupS9ilOOdcrcQ0VCSdLWmRpHxJt1ez3oWSTFJuMD1G0twKjzJJOZIyDpi/QdK9wTadJb0j6VNJ8yR9L5Zti7aZyzbyYf5Grhvag6YpSWGX45xztRKzv16SEoEJwDCgEJgtaaqZfX7AehnATcCs8nlm9izwbLD8OOBlM5sbLM6psO0c4KVg8pfAFDN7UFJv4FUgOwZNizoz4+43FtOueSpjBnYOuxznnKu1WPZUBgD5ZrbMzIqBF4DzKlnvd8CdwJ4q9jMaeP7AmZJ6Am2B94NZBpTfErEFsKb2pdet95ds4OOCTdxwWg/SkhPDLsc552otlqHSAVhVYbowmLefpP5AJzN7pZr9jKSSUCESNpPNzILp3wBjJRUS6aXcWMu665SZcde0xXRo2YSLTuwUdjnOOXdYYhkqqmSe7V8oJQD3ALdVuQNpILDLzBZUsngU3wyb0cATZtYR+B7wdPAaB+5znKQ8SXlFRUU1a0kMvfXFej5btYWbzuhBapL3Upxz8S2WoVIIVPzo3ZFvDkllAH2A6ZIKgEHA1PKD9YEDgwMASf2AJDObU2H2VcAUADObAaQBmQdua2aTzCzXzHKzsrJq066oKSsz7p62mC5tmvLD4zuGWotzzkVDLENlNtBTUldJKUQCYmr5QjPbamaZZpZtZtnATGC4meXB/p7MCCLHYg5U2XGWlcAZwbbHEAmV8Lsi1Xh94Vo+/2obN5/Zk+REP7vbORf/Ynb2l5mVSLoBeB1IBB4zs4WS7gDyzGxq9XtgCFBoZssqWXYRkSGuim4DHpZ0C5FhtssrHG+pd0qDXkqPts0Y3q/DwTdwzrk4ENMvRJjZq0QOmlec96sq1h16wPR0IkNila3brZJ5nwMn17LUOvfKvDUsWb+DCRcfT2JCZYefnHMu/viYSwhKSsu4980lHH1EBuf0OSLscpxzLmo8VELw0qerWb5hJ7cO60WC91Kccw2Ih0odKy4p4/63ltC3YwuG9W4XdjnOORdVHip1bEreKgo37+bWYb2QvJfinGtYPFTq0J59pTzwdj65XVpxaq9wvyPjnHOx4KFSh57/eCVrt+3h1rO8l+Kca5g8VOrI7uJSJryzlMHd2nBS92990d855xoED5U68tSMAjbs2MttZ/UKuxTnnIsZD5U6sGNvCQ+9u5RTe2WRm9067HKccy5mPFTqwOMfLGfzrn3eS3HONXgeKjG2ddc+Jr2/jGG929G3Y8uwy3HOuZjyUImxRz5YxvY9Jdw6zHspzrmGz0MlhjbtLOaxD5bz/b5HcsyRzQ++gXPOxTkPlRia+O5Sdu8r5ZYze4ZdinPO1QkPlRhZv30PT84o4PycDvRomxF2Oc45Vyc8VGLkwelL2Vdq3HSG91Kcc42Hh0oMfLV1N8/OXMmFx3ckOzM97HKcc67OeKjEwANv52MYN57RI+xSnHOuTnmoRNmqTbuYPHsVo07sTMdWTcMuxznn6lRMQ0XS2ZIWScqXdHs1610oySTlBtNjJM2t8CiTlCMp44D5GyTdW2E/F0n6XNJCSc/Fsm1Vuf+tJSQkiOtP816Kc67xSYrVjiUlAhOAYUAhMFvSVDP7/ID1MoCbgFnl88zsWeDZYPlxwMtmNjdYnFNh2znAS8HznsAvgJPNbLOktrFqW1WWFe3gpU9Xc/lJ2RzRIq2uX94550IXy57KACDfzJaZWTHwAnBeJev9DrgT2FPFfkYDzx84MwiRtsD7waxrgAlmthnAzNYfXvmH7r63lpCSmMB1Q7vX9Us751y9EMtQ6QCsqjBdGMzbT1J/oJOZvVLNfkZSSagQCZvJZmbBdC+gl6QPJc2UdHZlO5M0TlKepLyioqKatuWgFq/bztTP1nD5ydlkNkuN2n6dcy6exDJUKru1oe1fKCUA9wC3VbkDaSCwy8wWVLJ4FN8MmySgJzCUSOA8IulbV3A0s0lmlmtmuVlZ0bul771vLiY9JYlx3+kWtX0651y8iWWoFAKdKkx3BNZUmM4A+gDTJRUAg4Cp5QfrAwcGBwCS+gFJZjbngNd72cz2mdlyYBGRkIm5hWu28ur8tVx5SldapafUxUs651y9FMtQmQ30lNRVUgqRgJhavtDMtppZppllm1k2MBMYbmZ5sL8nM4LIsZgDVXac5V/AacG2mUSGw5ZFt0mVu2faYlo0SeaqU7rWxcs551y9FbNQMbMS4AbgdeALYIqZLZR0h6ThNdjFEKDQzCoLhov4dqi8DmyU9DnwDvAzM9tY+xbUzKcrN/PmF+sZN6QbLZokx/rlnHOuXtPXx7kbn9zcXMvLyzusfVzy6CwWrtnG+z8/jfTUmJ2h7Zxz9YakOWaWW9ky/0b9Yfh4+SbeX7KB607t7oHinHN4qNSamXHXG4vIykhl7KAuYZfjnHP1godKLX20dCOzlm/ihtN60CQlMexynHOuXvBQqQUz4y9vLKJ9izRGDeh08A2cc66R8FCphemLivh05RZuPKMnqUneS3HOuXIeKrWwfW8J/Tu35MITOoZdinPO1St+ylItDO/XnnP7HolU2ZVonHOu8fKeSi15oDjn3Ld5qDjnnIsaDxXnnHNR46HinHMuajxUnHPORY2HinPOuajxUHHOORc1HirOOeeiplHfT0VSEbCilptnAhuiWE6YvC31T0NpB3hb6qvDaUsXM8uqbEGjDpXDISmvqpvUxBtvS/3TUNoB3pb6KlZt8eEv55xzUeOh4pxzLmo8VGpvUtgFRJG3pf5pKO0Ab0t9FZO2+DEV55xzUeM9Feecc1HjoeKccy5qPFRqSFKBpPmS5krKC+a1ljRN0pLgZ6uw6zyYKtrxG0mrg3lzJX0v7DprQlJLSX+X9KWkLyQNjsf3BKpsS9y9L5KOqlDvXEnbJN0cb+9LNe2Iu/cEQNItkhZKWiDpeUlpkrpKmhW8J5MlpUTltfyYSs1IKgByzWxDhXl3ApvM7I+Sbgdamdn/C6vGmqiiHb8BdpjZX8KqqzYkPQm8b2aPBL8QTYH/jzh7T6DKttxMHL4v5SQlAquBgcD1xOH7At9qxxXE2XsiqQPwAdDbzHZLmgK8CnwPeMnMXpD0EPCZmT14uK/nPZXDcx7wZPD8SeD8EGtpVCQ1B4YAjwKYWbGZbSEO35Nq2hLvzgCWmtkK4vB9qaBiO+JVEtBEUhKRDyxfAacDfw+WR+098VCpOQPekDRH0rhgXjsz+wog+Nk2tOpqrrJ2ANwgaZ6kx+r70ESgG1AEPC7pU0mPSEonPt+TqtoC8fe+VDQKeD54Ho/vS7mK7YA4e0/MbDXwF2AlkTDZCswBtphZSbBaIdAhGq/noVJzJ5vZ8cA5wPWShoRdUC1V1o4Hge5ADpH/dHeFWF9NJQHHAw+aWX9gJ3B7uCXVWlVticf3BYBgCG848GLYtRyOStoRd+9JEHznAV2B9kA6kd//A0XlWIiHSg2Z2Zrg53rgn8AAYJ2kIwGCn+vDq7BmKmuHma0zs1IzKwMeJtK2+q4QKDSzWcH034n8YY6794Qq2hKn70u5c4BPzGxdMB2P7wsc0I44fU/OBJabWZGZ7QNeAk4CWgbDYQAdgTXReDEPlRqQlC4po/w5cBawAJgKXBasdhnwcjgV1kxV7Sj/ZQ9cQKRt9ZqZrQVWSToqmHUG8Dlx9p5A1W2Jx/elgtF8c8go7t6XwDfaEafvyUpgkKSmksTXvyvvABcG60TtPfGzv2pAUjcin+ohMlTxnJn9XlIbYArQmcgbN8LMNoVU5kFV046niXTnDSgAxpePf9dnknKAR4AUYBmRM3MSiKP3pFwVbbmf+HxfmgKrgG5mtjWYF1e/K1BlO+L1d+W3wEigBPgUuJrIMZQXgNbBvLFmtvewX8tDxTnnXLT48Jdzzrmo8VBxzjkXNR4qzjnnosZDxTnnXNR4qDjnnIsaDxXnnHNR46HiXD0i6QlJFx58TefqJw8V55xzUeOh4txBSMoObpz1cHCjozckNZE0XVJusE5mcK8aJF0u6V+S/i1puaQbJN0aXIF4pqTWNXzdEyS9G1xR+vUK1866RtJsSZ9J+kdw+Y0WityALSFYp6mkVZKSJXWX9Fqwn/clHR2sMyK4adNnkt6LyT+ea3Q8VJyrmZ7ABDM7FtgC/Ogg6/cBLiZywcHfA7uCKxDPAC492ItJSgb+ClxoZicAjwX7gciNlU40s37AF8BVwWVEPgNODdY5F3g9uIDgJODGYD8/Bf4WrPMr4LvBfoYfrCbnaiLp4Ks454hc5XVu8HwOkH2Q9d8xs+3AdklbgX8H8+cDfWvwekcRCaZpkWsAkkjkUusAfST9D9ASaAa8HsyfTOT6Tu8QuQfI3yQ1I3JF2heD/QCkBj8/BJ4I7gT4Ug1qcu6gPFScq5mKF9orBZoQuThfeW8/rZr1yypMl1Gz3zsBC81scCXLngDON7PPJF0ODA3mTwX+NxheOwF4m8i9M7aYWc6BOzGzayUNBL4PzJWUY2Yba1Cbc1Xy4S/naq+AyB9v+PoS4tGyCMiSNBgiw2GSjg2WZQBfBUNkY8o3MLMdwMfAfcArwX0/tgHLJY0I9iNJ/YLn3c1slpn9CtgAdIpyG1wj5KHiXO39BbhO0kdAZjR3bGbFRILqT5I+A+YSGcYC+G9gFjAN+PKATScDY4Of5cYAVwX7WUjkLoAAf5Y0X9IC4D0ix2ScOyx+6XvnnHNR4z0V55xzUeMH6p0LgaQJwMkHzL7PzB4Pox7nosWHv5xzzkWND38555yLGg8V55xzUeOh4pxzLmo8VJxzzkXN/w/Ek/7Ou614uAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "#train_means = grid_search.cv_results_['mean_train_score']\n",
    "#train_stds = grid_search.cv_results_['std_train_score']\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.47751952, -0.47896522, -0.47826173, -0.47833123])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  6.0min remaining:  3.0min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  8.6min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4,\n",
       "             param_grid={'min_child_samples': range(10, 50, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47476563714086817\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD6CAYAAACoCZCsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnO4GQCIQ1wYDikrIEGIEqeK3+vBeXYhdQqLW1rtUi3Kv2Vu/aq723rV28orS9VMUNQaxVKaXaRamghRJklcUiRQmgxCpRRJbA5/fHnAlDnJAJJJnt/Xw88kjOd86cfL6PA/PO+X7nzNfcHRERyTxZiS5AREQSQwEgIpKhFAAiIhlKASAikqEUACIiGUoBICKSoeIKADMba2YbzWyTmd12lP3Gm5mbWSjYzjWzh81sjZmtN7Pbo/Z90Mx2mtna4++GiIi0VE5zO5hZNjAdOB+oAZaZ2Tx3X9dovyJgCrA0qnkCkO/ug8ysEFhnZrPdfQvwEHAf8Ei8xXbr1s0rKiri3V1ERIDly5e/6+6ljdubDQBgBLDJ3TcDmNkc4BJgXaP97gTuAm6NanOgo5nlAB2A/cAHAO7+kplVtKQTFRUVVFdXt+QpIiIZz8zejNUezxBQH2Br1HZN0BZ98KFAubvPb/TcXwAfATuAt4Afuvt78RYdHPs6M6s2s+ra2tqWPFVERI4ingCwGG0Nnx9hZlnA3cAtMfYbARwEegP9gFvMrH9LCnT3Ge4ecvdQaeknrmBEROQYxTMEVAOUR22XAdujtouAgcBCMwPoCcwzs3HAl4Dn3P0AsNPMXgZCwOZWqF1ERI5DPFcAy4ABZtbPzPKAicC8yIPuXufu3dy9wt0rgCXAOHevJjzsc66FdQRGARtavRciItJizQaAu9cDk4HngfXAXHd/zczuCP7KP5rpQCdgLeEgmenuqwHMbDbwJ+BUM6sxs6uPox8iItJClkofBx0KhVzvAhIRaRkzW+7uocbtuhNYRCRDZUQAPL70LV56XW8hFRGJlvYBsL/+EI8teZNrH6lWCIiIREn7AMjLyeKxa0bSv7STQkBEJEraBwBAl455zApC4BqFgIgIkCEBAOEQePyakZykEBARATIoAABOUAiIiDTIqACAT4bAHxUCIpKhMi4A4HAInBxMDCsERCQTZWQAQDgEZikERCSDZWwAgEJARDJbRgcAfDIEFm7cmeiSRETaRcYHABwOgQHdO3Hdo8sVAiKSERQAAYWAiGQaBUCUkkKFgIhkDgVAIwoBEckUCoAYFAIikgkUAE1QCIhIulMAHEUkBE7p0YnrHlnOiwoBEUkjCoBmlBTm8djVIzmlZyeuVwiISBpRAMRBISAi6UgBEKdPhMAGhYCIpDYFQAuUFOYx6+pR4RB4VCEgIqlNAdBCxYW5CgERSQtxBYCZjTWzjWa2ycxuO8p+483MzSwUbOea2cNmtsbM1pvZ7S09ZjJqHAIvbHgn0SWJiLRYswFgZtnAdOACoBKYZGaVMfYrAqYAS6OaJwD57j4IGA5cb2YV8R4zmUVC4NSeRXz90VcVAiKScuK5AhgBbHL3ze6+H5gDXBJjvzuBu4C9UW0OdDSzHKADsB/4oAXHTGrFhbk8dvVIhYCIpKR4AqAPsDVquyZoa2BmQ4Fyd5/f6Lm/AD4CdgBvAT909/fiOWbUsa8zs2ozq66tTb4FWxQCIpKq4gkAi9HmDQ+aZQF3A7fE2G8EcBDoDfQDbjGz/s0d84hG9xnuHnL3UGlpaRzltr9ICJzWKxwCf1ivEBCR5BdPANQA5VHbZcD2qO0iYCCw0My2AKOAecFE8JeA59z9gLvvBF4GQnEcM+UUF+by6FXhELjhMYWAiCS/eAJgGTDAzPqZWR4wEZgXedDd69y9m7tXuHsFsAQY5+7VhId9zrWwjoTDYUNzx0xVCgERSSXNBoC71wOTgeeB9cBcd3/NzO4ws3HNPH060AlYS/hFf6a7r27qmMfRj6RRXJjLo5HhoMeWKwREJGmZe8yh96QUCoW8uro60WXEpe7jA1zxwFLW7/iAn315OOed3iPRJYlIhjKz5e4eatyuO4HbSHGH8JXA6b0660pARJKSAqANKQREJJkpANpYJAQqgxD4/TqFgIgkBwVAOyjukMsjQQjcMEshICLJQQHQThQCIpJsFADtqCEEehdzw6zl/E4hICIJpABoZ8UdcnnkqhFU9i7mRoWAiCSQAiABFAIikgwUAAkSfnfQ4RD47WtvJ7okEckwCoAE6lxwOAS+8firCgERaVcKgARTCIhIoigAkkAkBD6lEBCRdqQASBKdC3J5RCEgIu1IAZBEokPgxlkKARFpWwqAJBMJgYF9FAIi0rYUAEkoEgKDysIh8LxCQETagAIgSXUuyOXhq8Ih8A2FgIi0AQVAElMIiEhbUgAkuc4F4Y+NUAiISGtTAKSAokYh8NxahYCIHD8FQIqIDoHJjysEROT4KQBSSCQEBisERKQVKABSTFEwMXw4BHYkuiQRSVEKgBR0ZAisUAiIyDGJKwDMbKyZbTSzTWZ221H2G29mbmahYPtyM1sZ9XXIzKqCxy4zs9Vm9pqZ3dU63ckcCgEROV7NBoCZZQPTgQuASmCSmVXG2K8ImAIsjbS5+yx3r3L3KuAKYIu7rzSzrsAPgPPc/VNADzM7r1V6lEEiITCkvITJj6/gN2sUAiISv3iuAEYAm9x9s7vvB+YAl8TY707gLmBvE8eZBMwOfu4PvO7utcH274Evxl21NCgqyOWhr53BkPISbpqtEBCR+MUTAH2ArVHbNUFbAzMbCpS7+/yjHOcyDgfAJuA0M6swsxzgc0B5rCeZ2XVmVm1m1bW1tbF2yXgKARE5FvEEgMVo84YHzbKAu4FbmjyA2Uhgj7uvBXD394EbgCeARcAWoD7Wc919hruH3D1UWloaR7mZKXo4SCEgIvGIJwBqOPKv8zJge9R2ETAQWGhmW4BRwLzIRHBgIof/+gfA3X/l7iPd/dPARuAvLS9fonXKzzk8J6AQEJFmxBMAy4ABZtbPzPIIv5jPizzo7nXu3s3dK9y9AlgCjHP3ami4QphAeO6ggZl1D76fANwI3N8K/cl4kRCoUgiISDOaDQB3rwcmA88D64G57v6amd1hZuPi+B1nAzXuvrlR+z1mtg54Gfieu7/ewtqlCZEQGKoQEJGjMHdvfq8kEQqFvLq6OtFlpIzd++q58sE/s2LrLu6dNJQLB/VKdEkikgBmttzdQ43bdSdwGuuUn8NDwZXATbNXsEBXAiISRQGQ5iIhMKyvQkBEjqQAyACd8nOY+TWFgIgcSQGQIRqHwK9XKwREMp0CIINEh8CUOQoBkUynAMgwCgERiVAAZKBO+Tk8FBUC81dvb/5JIpJ2FAAZqmNUCEyds1IhIJKBFAAZLBICw/ueoBAQyUAKgAzXMT+HmV87oyEEfrVKISCSKRQAckQI/OMTCgGRTKEAECAqBE5UCIhkCgWANOiYn8PMKxUCIplCASBHUAiIZA4FgHxCdAhMnbNCISCSphQAElMkBEIVXRQCImlKASBNCt8ncDgE5ikERNKKAkCOqjDvcAj8o0JAJK0oAKRZkRA4QyEgklYUABKXwrzwfQIKAZH0oQCQuDUOgWdXbkt0SSJyHBQA0iKREBjRrwv/9MRKhYBIClMASIsV5uXw4JUKAZFUpwCQY6IQEEl9cQWAmY01s41mtsnMbjvKfuPNzM0sFGxfbmYro74OmVlV8NgkM1tjZqvN7Dkz69Y6XZL2EgmBkf26KgREUlCzAWBm2cB04AKgEphkZpUx9isCpgBLI23uPsvdq9y9CrgC2OLuK80sB7gH+Iy7DwZWA5Nbo0PSvgrzcnjgypBCQCQFxXMFMALY5O6b3X0/MAe4JMZ+dwJ3AXubOM4kYHbwswVfHc3MgM6A3leYohqHwDMrFAIiqSCeAOgDbI3argnaGpjZUKDc3ecf5TiXEQSAux8AbgDWEH7hrwQeiPUkM7vOzKrNrLq2tjaOciURokPg5rkKAZFUEE8AWIw2b3jQLAu4G7ilyQOYjQT2uPvaYDuXcAAMBXoTHgK6PdZz3X2Gu4fcPVRaWhpHuZIo0XMCCgGR5BdPANQA5VHbZRw5XFMEDAQWmtkWYBQwLzIRHJjI4eEfgCoAd3/D3R2YC5zZ4uol6XTIy1YIiKSIeAJgGTDAzPqZWR7hF/N5kQfdvc7du7l7hbtXAEuAce5eDQ1XCBMIzx1EbAMqzSzyJ/35wPrj7o0khcYh8PSKmkSXJCIxNBsA7l5P+B06zxN+kZ7r7q+Z2R1mNi6O33E2UOPum6OOuR34L+AlM1tN+Irgf46lA5KcIiEwqn9Xbpm7SiEgkoQsPAKTGkKhkFdXVye6DGmBj/cf5OqHl7Fk89/40aVD+PzQskSXJJJxzGy5u4cat+tOYGlTHfKyeeCruhIQSUYKAGlzkRD49ElduVkhIJI0FADSLjrkZXP/V87gTIWASNJQAEi7UQiIJBcFgLSrxiHwg+c3UH/wUKLLEslICgBpd5E5gUuHlzP9xTeY9PMlbN/1caLLEsk4CgBJiILcbL4/fjD3TKxi3fYPuHDaIn6/7p1ElyWSURQAklCXVPVh/pQx9CnpwDWPVHPn/HXsr9eQkEh7UABIwvXr1pFf3ngmV55ZwQOL/8r4n73CW3/bk+iyRNKeAkCSQn5ONt8e9yl+9uVhbHn3Iy6atoj5q7VEhEhbUgBIUhk7sBe/njKGk7p3YvLjK/jXp9ew98DBRJclkpYUAJJ0yrsU8uTXP831f9efWUvf4nPTX2bTzt2JLksk7SgAJCnlZmdx+wWnM/NrZ7Dzw3189t7FPLVcN46JtCYFgCS1z5zanQVTxjC4rJhbnlzFzXNX8tG++kSXJZIWFACS9HoWF/D4taOYet4Anl6xjc/et5j1Oz5IdFkiKU8BICkhO8v4p/NPYdY1I9m9t55Lpr/MY0veJJXWsxBJNgoASSlnntSNBVPHMKp/V/7tmbVMfnwFH+w9kOiyRFKSAkBSTrdO+Tx05Rl8a+xpPPfa21w8bTGrtu5KdFkiKUcBICkpK8u44ZyTmHv9KA4ecsb/7BXuX7RZQ0IiLaAAkJQ2/MQu/HrKaM45tTvf+fV6rn2kmvc/2p/oskRSggJAUl5JYR4zrhjOtz9byUuvv8uF0xaxbMt7iS5LJOkpACQtmBlXntWPp244k7ycLCbOWML0Fzdx6JCGhESaogCQtDKorJj5N43mwkG9+MHzG/nqzD9T++G+RJclkpQUAJJ2igpymTaxiu99YRB//ut7XHDPIhb/5d1ElyWSdOIKADMba2YbzWyTmd12lP3Gm5mbWSjYvtzMVkZ9HTKzKjMratT+rpn9b2t1SsTMmDiiL/Mmj6akMJcrHlzKj367UesPi0RpNgDMLBuYDlwAVAKTzKwyxn5FwBRgaaTN3We5e5W7VwFXAFvcfaW7fxhpDx57E/hl63RJ5LBTexYxb/JZTBhexr0vbOJLP1/KjjqtPywC8V0BjAA2uftmd98PzAEuibHfncBdwN4mjjMJmN240cwGAN2BRXFVLNJChXk53DV+CHdfNoS12+u48J5FvLBB6w+LxBMAfYCtUds1QVsDMxsKlLv7/KMc5zJiBADhYHjCm7iDx8yuM7NqM6uura2No1yR2D4/tIz5N42mV3EHrnqomu9o/WHJcPEEgMVoa3ixNrMs4G7gliYPYDYS2OPua2M8PJHYwRD+Re4z3D3k7qHS0tI4yhVpWv/STvzyxjP5yqdP5P7Ff2XCz15h63taf1gyUzwBUAOUR22XAdGLtRYBA4GFZrYFGAXMi0wEB2K+yJvZECDH3Ze3sG6RY1aQm80dlwzkZ18exuZ3P+LCaYtYsGZHossSaXfxBMAyYICZ9TOzPMIv5vMiD7p7nbt3c/cKd68AlgDj3L0aGq4QJhCeO2gs5ryASHsYO7AXC6aM4aTSTtw461X+7RmtPyyZpdkAcPd6YDLwPLAemOvur5nZHWY2Lo7fcTZQ4+6bYzx2KQoASaCG9YfP7s9jS97i8z95hTdqtf6wZAZLpU9PDIVCXl1dnegyJE29uGEnN89dyb76Q3zncwP5wrCyRJck0irMbLm7hxq3605gkcBnTuvOb6aezcA+xdw8dxW3PrmKPfu1/rCkLwWASJSexQU8fs1Ippw3gKdereGz9y5mw9taf1jSkwJApJGc7CxuPv8UZl09kg/21nPJfS/z+NK3tNiMpB0FgEgTzjy5GwumjGFEvy78y9NruGn2Cj7U+sOSRhQAIkdRWpTPw18bwT+PPZXfrH2bi6YtZnWN1h+W9KAAEGlGVpZx4zkn88R1o6g/eIgv/vQVHlz8Vw0JScpTAIjEKVTRhQVTx/B3p3TnjvnruPaR5ezao/WHJXUpAERaoKQwj59/ZTj/cXElf3x9Jxfes4hqrT8sKUoBINJCZsZVo8PrD+dkZ3GZ1h+WFKUAEDlGg8tKmD9lNBcM7Kn1hyUlKQBEjkPnglzunTSU7wbrD184bRGvbNL6w5IaFAAix8nMmDSiL89OPovOBTlc/sBSfqz1hyUFKABEWslpPTvzq5tGM35YGdNe2MSX7l/K23VNrZAqkngKAJFWVJiXww8mDOHHlw5h7bY6LrjnJa0/LElLASDSBr4wrIxf3TSansH6w/+zYL3WH5akowAQaSMnlXbi6RvP5IpRJzLjpc1M+L8/af1hSSoKAJE2VJCbzZ2fG8hPLh/G5p27uXDaIn6j9YclSSgARNrBhYN6sWDqGPqXduKGWa/y78+s1frDknAKAJF2Ut6lkCev/zTXjunHo0ve5PM/eYXNWn9YEkgBINKO8nKy+NeLKnnwyhBv133Mxfcu5ukVNYkuSzKUAkAkAc49rQcLpo5hYO9i/umJVXxT6w9LAigARBKkV3EHHr92JFPOPZlfvFrDuPteZuPbHya6LMkgCgCRBMrJzuLmvz+Vx64eya49Bxh332Jm/1nrD0v7UACIJIGzTu7Gb6aG1x++/ZdrmDJnpdYfljYXVwCY2Vgz22hmm8zstqPsN97M3MxCwfblZrYy6uuQmVUFj+WZ2Qwze93MNpjZF1unSyKpKbL+8Df/4VQWrNnBxfcuZk1NXaLLkjTWbACYWTYwHbgAqAQmmVlljP2KgCnA0kibu89y9yp3rwKuALa4+8rg4X8Fdrr7KcFx/3i8nRFJdVlZxjc+E15/eH/9Ib7w05eZ+bLWH5a2Ec8VwAhgk7tvdvf9wBzgkhj73QncBTT18YeTgNlR21cB3wVw90Purg9RFwmEKrqwYMoY/u6UUv7rV+u47lGtPyytL54A6ANsjdquCdoamNlQoNzd5x/lOJcRBICZlQRtd5rZq2b2pJn1iPUkM7vOzKrNrLq2tjaOckXSwwkd8/j5V0L8+8WVLNy4k4umLWb5m1p/WFpPPAFgMdoarkfNLAu4G7ilyQOYjQT2uPvaoCkHKANedvdhwJ+AH8Z6rrvPcPeQu4dKS0vjKFckfZgZVwfrD2dnGZf+3xJ+uvANrT8srSKeAKgByqO2y4DtUdtFwEBgoZltAUYB8yITwYGJHDn88zdgD/B0sP0kMKxFlYtkkMj6w2MH9uT7z23gyoeW8e5urT8sxyeeAFgGDDCzfmaWR/jFfF7kQXevc/du7l7h7hXAEmCcu1dDwxXCBMJzB5HnOPAr4Jyg6Txg3fF3RyR9dS7I5b5JQ/nvzw9k6ea/ccE9i3jlDU2dybFrNgDcvR6YDDwPrAfmuvtrZnaHmY2L43ecDdS4++ZG7d8Cvm1mqwm/Q6jJISQRCTMzLh95Is98I1h/+P6l/Ph3r3NQQ0JyDCyV3l4WCoW8uro60WWIJIWP9tXzH8++xlOv1jCyXxfumTiUnsUFiS5LkpCZLXf3UON23QkskqI65ufwo0uH8KMJQ1izrY4Lpy3ixY07E12WpBAFgEiK++LwMuZNHk33ony+NnMZ312wngMHtf6wNE8BIJIGTu7eiWe+cRZfHtWX/3tpM5dq/WGJgwJAJE0U5Gbznc8NYvqXhrHpnd1cNG0Rz63V+sPSNAWASJq5aHAvfj1lDP26deTrj73Kfz6r9YclNgWASBrq27WQJ79+JteM7sfDf3qTL/70Ff767keJLkuSjAJAJE3l5WTxbxdX8sBXQ2zb9TEXT1vEsyu3JbosSSIKAJE0d97pPVgwZQyVvTszdc5KvvWL1Xy8X0NCogAQyQi9Szow+9pRTP7MycxdvpVx9y3m9Xe0/nCmUwCIZIic7Cxu/YdTefSqkbwfrD/8xDKtP5zJFAAiGWb0gG4smDqa4SeewLeeWsNUrT+csRQAIhmoe1EBj1w1klv//hTmr97OZ+9dzNptWn840ygARDJUdpYx+dwBPHH9p9lXf4gv/OQVHtL6wxlFnwYqIrz/0X5ufXIVf9iwk97FBVT1LWFwWQlDykoYVFZMp/ycRJcox6GpTwPVWRURTuiYx/1fDfGL5TX88fVaVtfUsWDN2wCYwcmlncKBUF7MkLISTutVRH5OdoKrluOlKwARiem9j/azumYXq7bWhb/X7OLd3fsByMvO4vReRQwuK2FwWTFV5SX0L+1EdlasJcQl0Zq6AlAAiEhc3J3tdXtZvXUXK2t2sXprHWu21bF7Xz0AHfOyGVQWvkKIXC30KemAmUIh0TQEJCLHxczoU9KBPiUduGBQLwAOHXI2v7ubVVvrWFWzi1U1dcx8eQv7g/UIunbMY0h5+CphSHC10LVTfiK7IVEUACJyzLKyjJO7F3Fy9yK+OLwMgP31h9jw9gesqqlj1dZdrK7ZxYsbdxIZbCg7oQNDyksYUlbM4LISBvUppqMmmRNCQ0Ai0uZ276tn7ba6hjmFVTW7qHn/YwCyLLygzZCyEgaXl1BVVsKpPYvIy9G71FuLhoBEJGE65ecwqn9XRvXv2tD2t937WF0TDB1t3cULG3by5PIaIJhk7t2ZquAqYUh5Cf27dSRLk8ytSlcAIpIU3J2a9z8+IhTWbqvjo+CTS4vycxjYp7hh+GhIeQm9igs0yRwHXQGISFIzM8q7FFLepZCLBocnmQ8ect6o3c2qreG3oa6uqeOBxZs5cDD8h2u3TvkNYRCZaD6hY14iu5FSFAAikrSys4xTehRxSo8iJoTKAdhXf5D1Oz5kdc0uVm4Nh8ILUZPMfbsUHjHJPLBPZwrz9FIXS1xDQGY2FrgHyAbud/fvNbHfeOBJ4Ax3rzazy4FvRu0yGBjm7ivNbCHQC/g4eOzv3X3n0erQEJCIxPLh3gOs2VYXHj4KQmHbrsOTzKf0KAommcNXCaf2LCI3O3MmmY/5RjAzywZeB84HaoBlwCR3X9dovyLg10AeMNndqxs9Pgh41t37B9sLgVsb73c0CgARiVfth/uCO5gPvx31/T3hj73Oz8misndnhgQ3rA0uK6Ff1/SdZD6eOYARwCZ33xwcaA5wCbCu0X53AncBtzZxnEnA7LgrFhE5DqVF+Zx3eg/OO70HEJ5k3vrex8FcQvjtqE8s28pDr2wBoKggJ+qGtXAw9Oyc3pPM8QRAH2Br1HYNMDJ6BzMbCpS7+3wzayoALiMcHNFmmtlB4CngOx7jcsTMrgOuA+jbt28c5YqIfJKZ0bdrIX27FvLZIb0BqD94iE21u1ndcCfzLma8tJn6Q+GXou5F+QwuK6EquEoYXFZMSWH6TDLHEwCx4q/hhdrMsoC7gSubPIDZSGCPu6+Nar7c3bcFQ0dPAVcAj3ziF7nPAGZAeAgojnpFROKSk53FaT07c1rPzlx6RniSee+Bg6zb8QGrtwbDRzW7+P36dxqeU9G1sOHehCFlxXyqdzEd8lLzk1HjCYAaoDxquwzYHrVdBAwEFgaXSj2BeWY2Lmp8fyKNhn/cfVvw/UMze5zwUNMnAkBEpD0V5GYzrO8JDOt7QkNb3ccHWLvt8P0Jy7a8x7xV4ZfByDuVot+OekqP1JhkjicAlgEDzKwfsI3wi/mXIg+6ex3QLbLdeHI3uEKYAJwdtU8OUOLu75pZLnAx8Pvj7o2ISBso7pDLWSd346yTG17q2PnBXlbV1DW8HfU3a99mzrLwaHl+ThYD+xQ3fFT24LISKroWJt18QrMB4O71ZjYZeJ7w20AfdPfXzOwOoNrd5zVziLOBmsgkciAfeD548c8m/OL/82PqgYhIAnTvXMD5lQWcX3l4kvmt9/Y03JuwausuZv/5LWa+vAUIh8jgsuKGieYh5SX06FyQwB7ooyBERNpM/cFDvP7O7iPejrrxnQ85GEwy9+xcEA6E8sPLbxZ3yG31OvRRECIi7SwnO3y/QWXvzkwcEW77eP9B1u2oi1pprY7frjs8ydy/W8fgSiF8lfCp3p0pyG2bSWYFgIhIO+qQl83wE7sw/MQuDW11ew6wetvhoaM/bf4bz6wMTzLnZBmn9ixi1jUjW/0tqAoAEZEEKy7MZcyAUsYMKG1oe7tub8NNa5t27m6ToSEFgIhIEupZXEDP4p78w6d6ttnvSP43qoqISJtQAIiIZCgFgIhIhlIAiIhkKAWAiEiGUgCIiGQoBYCISIZSAIiIZKiU+jA4M6sF3jzGp3cD3m3FchIpXfqSLv0A9SVZpUtfjrcfJ7p7aePGlAqA42Fm1bE+DS8VpUtf0qUfoL4kq3TpS1v1Q0NAIiIZSgEgIpKhMikAZiS6gFaULn1Jl36A+pKs0qUvbdKPjJkDEBGRI2XSFYCIiERRAIiIZKi0DAAze9DMdprZ2qi2Lmb2OzP7S/D9hETWGI8m+vFtM9tmZiuDrwsTWWO8zKzczF40s/Vm9pqZTQ3aU+q8HKUfKXdezKzAzP5sZquCvvxX0N7PzJYG5+QJM2vddQjbwFH68pCZ/TXqvFQlutZ4mFm2ma0ws/nBdpuck7QMAOAhYGyjttuAP7j7AOAPwXaye4hP9gPgbnevCr4WtHNNx6oeuMXdTwdGAd8ws0pS77w01Q9IvfOyDzjX3YcAVcBYMxsFfJ9wXwYA7wNXJ7DGeDXVF4BvRp2XlYkrsUWmAuujttvknKRlALj7S8B7jZovAR4Ofn4Y+Fy7FnUMmuhHSnL3He7+avDzh4T/cfchxc7LUVOs6GAAAAJcSURBVPqRcjxsd7CZG3w5cC7wi6A96c8JHLUvKcfMyoCLgPuDbaONzklaBkATerj7Dgj/Jwa6J7ie4zHZzFYHQ0RJPWQSi5lVAEOBpaTweWnUD0jB8xIMNawEdgK/A94Adrl7fbBLDSkScI374u6R8/LfwXm528zyE1hivP4X+GfgULDdlTY6J5kUAOnip8BJhC9zdwA/Smw5LWNmnYCngH909w8SXc+xitGPlDwv7n7Q3auAMmAEcHqs3dq3qmPTuC9mNhC4HTgNOAPoAnwrgSU2y8wuBna6+/Lo5hi7tso5yaQAeMfMegEE33cmuJ5j4u7vBP/QDwE/J/yfNiWYWS7hF81Z7v7LoDnlzkusfqTyeQFw913AQsLzGiVmlhM8VAZsT1RdxyKqL2ODITt3933ATJL/vJwFjDOzLcAcwkM//0sbnZNMCoB5wFeDn78KPJvAWo5Z5MUy8HlgbVP7JpNgHPMBYL27/zjqoZQ6L031IxXPi5mVmllJ8HMH4P8RntN4ERgf7Jb05wSa7MuGqD8ujPC4eVKfF3e/3d3L3L0CmAi84O6X00bnJC3vBDaz2cA5hD9C9R3gP4FngLlAX+AtYIK7J/UEaxP9OIfwMIMDW4DrI2PoyczMRgOLgDUcHtv8F8Lj5ylzXo7Sj0mk2Hkxs8GEJxSzCf8xONfd7zCz/oT/+uwCrAC+HPwFnbSO0pcXgFLCwygrga9HTRYnNTM7B7jV3S9uq3OSlgEgIiLNy6QhIBERiaIAEBHJUAoAEZEMpQAQEclQCgARkQylABARyVAKABGRDPX/AW5wbTNnuIaIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "#train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "#train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.48075808, -0.47768608, -0.4754073 , -0.47476564])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 行采样参数 sub_samples/bagging_fraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  8.2min remaining:  2.0min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 10.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=30,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4,\n",
       "             param_grid={'subsample': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "subsample_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( subsample = subsample_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4754072980958521\n",
      "{'subsample': 0.7}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV9f3H8dcnkxVmwpAVdggRg4T5s46iFaoyFAVEC4hibSlttUOrdYCrtpXa1sFSURFE60BqpS6sg5UoIGGGABJm2Akh+/P7456EC1k3kOTcm3yej8d9cM/+nGu873vO95zvEVXFGGOM8RbkdgHGGGP8j4WDMcaYEiwcjDHGlGDhYIwxpgQLB2OMMSWEuF1AVYiMjNTo6Gi3yzDGmICSlJR0SFWjSptWK8IhOjqaxMREt8swxpiAIiK7yppmp5WMMcaUYOFgjDGmBAsHY4wxJVg4GGOMKcHCwRhjTAkWDsYYY0qwcDDGGFNCnQ6HLfszePI/m7Fuy40x5kx1Ohy+SjnEC59v54Pv9rtdijHG+BWfwkFEhorIFhFJEZF7y5lvtIioiCQ4w+NFZK3Xq1BE4p1pY0RkvYgki8hTXusIF5E3nG2tEpHo89vFsv1kUEd6XdCY6UuTycjOq67NGGNMwKkwHEQkGHgWGAbEAuNEJLaU+SKAacCqonGqukBV41U1HrgV2Kmqa0WkBfBnYIiq9gJaicgQZ7HJwFFV7QrMBP50XntYjpDgIB4bdSEHM3KY+dG26tqMMcYEHF+OHPoDKaqaqqq5wCJgRCnzzQCeArLLWM84YKHzvjOwVVXTneGPgRuc9yOA+c77t4AhIiI+1HlO4ts35eb+HXj56x1s2HO8ujZjjDEBxZdwaAvs9hpOc8YVE5E+QHtVXVrOesZwOhxSgBgRiRaREGAk0P7s7alqPnAcaHH2ykRkiogkikhienr62ZMr5XdXx9C8YRgPvLuBwkJrnDbGGF/CobRf7cXfoCIShOf0zz1lrkBkAJClqhsAVPUocBfwBvAFsBPI92V7xSNUZ6tqgqomREWV2uOsz5o0COX+a3qydvcxFq75/rzWZYwxtYEv4ZDG6V/1AO2AvV7DEUAcsFxEdgIDgSVFjdKOsZw+agBAVd9X1QGqOgjYAhSd9C/ennNU0QQ44usOnauR8W0Z1LkFf/rPZg5l5lT35owxxq/5Eg5rgG4i0klEwvB80S8pmqiqx1U1UlWjVTUaWAkMV9VEKD6yuBFPW0UxEWnp/NsM+Bkw15m0BJjgvB8NfKo1cCOCiDBjZByn8gp4/INN1b05Y4zxaxWGg3PefyqwDNgELFbVZBGZLiLDfdjGpUCaqqaeNf4ZEdkIfAU8qapbnfHzgBYikgLcDZR56WxV69qyEXde2oW3v9nDiu2Ha2qzxhjjd6Q23B2ckJCgVfUkuOy8Aq6a+TnhIcF8MO0HhIXU6fsEjTG1mIgkqWpCadPsm+8s9UKDmT48jpSDmcz54uyDHWOMqRssHEpxRUxLhvZqzd8/2cbuI1lul2OMMTXOwqEMDw2PJSRIeGhJsnXMZ4ypcywcytCmSX1+fVV3Pt18kGXJB9wuxxhjapSFQzkmDo4mpnUEj7yfzMmc/IoXMMaYWsLCoRyejvni2Hc8m799vLXiBYwxppawcKhA347NGde/PS9+tZNN+064XY4xxtQICwcf/H5oDE3qh1rHfMaYOsPCwQdNG4Rx37AYknYd5c2k3RUvYIwxAc7CwUej+7ajf3RznvjPZo6czHW7HGOMqVYWDj4SER4dFUdmdj5PWMd8xphazsKhErq3iuD2H3TmzaQ01uys9l7EjTHGNRYOlTRtSFfaNq3P/e98R15BodvlGGNMtbBwqKQGYSE8MrwXWw9kMu/LHW6XY4wx1cLC4RxcGduKq2Jb8czH20g7ah3zGWNqHwuHc/Tw8F4APPL+RpcrMcaYqmfhcI7aNq3PL6/sxkcbD/DRRuuYzxhTu1g4nIfJl3Sie6tGPLwkmaxc65jPGFN7+BQOIjJURLaISIqIlPlMZxEZLSIqIgnO8HgRWev1KhSReGfaOBH5TkTWi8iHIhLpjH9YRPZ4LfPjqtjR6hAaHMRjoy5kz7FT/P2TFLfLMcaYKlNhOIhIMPAsMAyIBcaJSGwp80UA04BVReNUdYGqxqtqPHArsFNV14pICPAMcIWq9gbWA1O9VjezaDlV/eA89q/a9Ytuzo192zH3i1S2HshwuxxjjKkSvhw59AdSVDVVVXOBRcCIUuabATwFZJexnnHAQue9OK+GIiJAY2BvZQr3J/f9uCeN6oXwwDsb7KlxxphawZdwaAt49zaX5owrJiJ9gPaqurSc9YzBCQdVzQPuAr7DEwqxwDyveac6p5teFJFmpa1MRKaISKKIJKanp/uwG9WneUNPx3yrdx7hraQ0V2sxxpiq4Es4SCnjin8ei0gQMBO4p8wViAwAslR1gzMciicc+gAX4DmtdJ8z+/NAFyAe2Af8tbR1qupsVU1Q1YSoqCgfdqN63di3PX07NuOJ/2zmqHXMZ4wJcL6EQxrQ3mu4HWeeAooA4oDlIrITGAgsKWqUdozl9Ckl8Hzxo6rb1XMeZjEw2Bl3QFULVLUQmIPntJbfCwoSHh0Zx/FTeTy1bLPb5RhjzHnxJRzWAN1EpJOIhOH5ol9SNFFVj6tqpKpGq2o0sBIYrqqJUHxkcSOetooie4BYESn6yX8VsMmZv43XfKOADee0Zy7o2aYxky/pxMLVu0naZR3zGWMCV4XhoKr5eK4kWobnC3yxqiaLyHQRGe7DNi4F0lQ11Wude4FHgP+JyHo8RxKPO5OfKrrEFbgC+HWl9shlvxzSjQua1OP+dzaQbx3zGWMClNSGq2sSEhI0MTHR7TKKfbhhPz99LYkHrunJ7T/o7HY5xhhTKhFJUtWE0qbZHdLV4OperfhhTEue/mgre4+dcrscY4ypNAuHaiAiPDK8F4WqTLeO+YwxAcjCoZq0b96AX/ywGx8m7+ezzQfdLscYYyrFwqEa3fGDznRt2YgHl2zgVG6B2+UYY4zPLByqUVhIEDNGxLH7yCn++dk2t8sxxhifWThUs0FdWnD9xW2Z/b9UUg5ax3zGmMBg4VAD/vDjnjQIC+GBd61jPmNMYLBwqAGRjcL53dAerEw9wrtr97hdjjHGVMjCoYaM69eB+PZNeXTpJo5n5bldjjHGlMvCoYYEBQmPjYrjaFaudcxnjPF7Fg41qNcFTZg4uBOvr/6etbuPuV2OMcaUycKhht39o+60jAjn/ne+s475jDF+y8KhhjUKD+Gh63qRvPcEr6zY5XY5xhhTKgsHFwyLa81l3aN4+qOtHDhR1iO3jTHGPRYOLhARpo/oRV5BIdOXWsd8xhj/Y+Hgko4tGvLzK7ry7/X7+HxrutvlGGPMGSwcXHTnZZ3pHNmQB9/bQHaedcxnjPEfFg4uCg8J5tGRcew6nMVzy7e7XY4xxhTzKRxEZKiIbBGRFBG5t5z5RouIikiCMzxeRNZ6vQpFJN6ZNq7oWdEi8qGIRDrjm4vIRyKyzfm3WVXsqL8a3DWSEfEX8MLy7aSmZ7pdjjHGAD6Eg4gEA88Cw4BYYJyIxJYyXwQwDVhVNE5VF6hqvKrGA7cCO1V1rYiEAM8AV6hqb2A9MNVZ7F7gE1XtBnziDNdq91/Tk/DQIP74nnXMZ4zxD74cOfQHUlQ1VVVzgUXAiFLmmwE8BZR1beY4YKHzXpxXQxERoDGw15k2ApjvvJ8PjPShxoDWMqIev7u6B1+lHGbJur0VL2CMMdXMl3BoC+z2Gk5zxhUTkT5Ae1VdWs56xuCEg6rmAXcB3+EJhVhgnjNfK1Xd58y3D2hZ2spEZIqIJIpIYnp64F/tc/OAjvRu14RH/72JE9nWMZ8xxl2+hIOUMq743IeIBAEzgXvKXIHIACBLVTc4w6F4wqEPcAGe00r3+V42qOpsVU1Q1YSoqKjKLOqXgoOEx0ZeyOHMHP66bIvb5Rhj6jhfwiENaO813I7Tp4AAIoA4YLmI7AQGAkuKGqUdYzl9SgkgHkBVt6vnJPtiYLAz7YCItAFw/j3o894EuAvbNeEng6J5ZeUu1qdZx3zGGPf4Eg5rgG4i0klEwvB80S8pmqiqx1U1UlWjVTUaWAkMV9VEKD6yuBFPW0WRPUCsiBT95L8K2OS8XwJMcN5PAN47pz0LUHf/qDuRjcK5/50NFBRa47Qxxh0VhoOq5uO5kmgZni/wxaqaLCLTRWS4D9u4FEhT1VSvde4FHgH+JyLr8RxJPO5MfhK4SkS24QmNJyuzQ4Gucb1Q/nhtLN/tOc6CVdYxnzHGHVIbLp1MSEjQxMREt8uoMqrKrfNWs273MT75zWW0jKjndknGmFpIRJJUNaG0aXaHtB8SEWaMjCOnoJBHl26qeAFjjKliFg5+qlNkQ+66rAtL1u3ly22H3C7HGFPHWDj4sbsu70J0iwb80TrmM8bUMAsHP1YvNJjpI+LYcegksz5PrXgBY4ypIhYOfu7S7lFc27sNzy5PYeehk26XY4ypIywcAsAfr40lLDiIB5ckW8d8xpgaYeEQAFo1rsc9P+rO/7am88F3+90uxxhTB1g4BIhbB3ak1wWNeeT9ZDKsYz5jTDWzcAgQIcFBPDbqQtIzc3j6o61ul2OMqeUsHAJIfPumjB/Qgflf72TDnuNul2OMqcUsHALMb6+OoXnDMO5/1zrmM8ZUHwuHANOkfigPXBPLut3HWLj6e7fLMcbUUhYOAWhE/AUM7tKCpz7cTHpGjtvlGGNqIQuHACQiTB8Rx6m8Ap74wDrmM8ZUPQuHANW1ZSPuvLQLb3+7h6+3W8d8xpiqZeEQwKb+sCsdmjfgj+9uIDe/0O1yjDG1iIVDAKsXGswjI3qxPf0kc76wjvmMMVXHwiHAXdGjJcPiWvP3T7bx/eEst8sxxtQSPoWDiAwVkS0ikiIi95Yz32gRURFJcIbHi8har1ehiMSLSMRZ4w+JyN+cZSaKSLrXtNurZldrrweviyUkSHhoyQbrmM8YUyUqDAcRCQaeBYYBscA4EYktZb4IYBqwqmicqi5Q1XhVjQduBXaq6lpVzSga70zbBbzttbo3vKbPPa89rAPaNKnPr6/qzmdb0lmWbB3zGWPOny9HDv2BFFVNVdVcYBEwopT5ZgBPAdllrGccsPDskSLSDWgJfOFTxaZUEwdH07NNYx55fyMnc/LdLscYE+B8CYe2wG6v4TRnXDER6QO0V9Wl5axnDKWEA57QeEPPPB9yg4isF5G3RKR9aSsTkSkikigiienp6T7sRu0WEhzEoyPj2Hc8m799bB3zGWPOjy/hIKWMK/4iF5EgYCZwT5krEBkAZKnqhlImj+XM0HgfiFbV3sDHwPzS1qmqs1U1QVUToqKiKt6LOqBvx2aM69+BF7/ayaZ9J9wuxxgTwHwJhzTA+9d7O2Cv13AEEAcsF5GdwEBgSVGjtOPsAABARC4CQlQ1qWicqh5W1aI+IeYAfX2o0Th+P7QHTeuHcv8731FoHfMZY86RL+GwBugmIp1EJAzPF/2SoomqelxVI1U1WlWjgZXAcFVNhOIjixvxtFWcrUQ7hIi08RocDlj/EJXQtEEY9/24J998f4zFibsrXsAYY0pRYTioaj4wFViG54t6saomi8h0ERnuwzYuBdJUtbS7tG6i5BHFNBFJFpF1eK5+mujDNoyXGy5uS/9OzXnyw80czrSO+YwxlSe14br4hIQETUxMdLsMv7LtQAbDnvmCkX3a8pcbL3K7HGOMHxKRJFVNKG2a3SFdS3VrFcEdl3bmraQ0Vu844nY5xpgAY+FQi037YTfaNq3PA+9+Zx3zGWMqxcKhFqsfFswjw3ux9UAm877c4XY5xpgAYuFQy10Z24ofxbbi759sI+2odcxnjPGNhUMd8NDwXgA8vGSjy5UYYwKFhUMd0LZpfX51ZTc+3nSA/1rHfMYYH1g41BG3XdKJHq0ieOT9jWTlWsd8xpjyWTjUEaHBQTw6Ko49x07xzCfb3C7HGOPnLBzqkH7RzbkpoR3zvtjBlv0ZbpdjjPFjFg51zL3DetKoXggPvGsd8xljymbhUMc0bxjGH4b1ZM3Oo7z1TZrb5Rhj/JSFQx00um87Ejo244kPNnH0ZK7b5Rhj/JCFQx0UFCQ8OiqOE9n5/OnDzW6XY4zxQxYOdVRM68ZMvqQTi9bsJmmXdcxnjDmThUMd9ssh3bigST3uf2cDeQXWMZ8x5jQLhzqsYXgIDw3vxeb9Gbz81U63yzHG+BELhzruR7GtGBLTkpkfb2XvsVNul2OM8RMWDnWciPDw8F4UqvLI+8lul2OM8RM+hYOIDBWRLSKSIiL3ljPfaBFREUlwhseLyFqvV6GIxItIxFnjD4nI35xlwkXkDWdbq0Qkuip21JStffMGTBvSjWXJB/h08wG3yzHG+IEKw0FEgoFngWFALDBORGJLmS8CmAasKhqnqgtUNV5V44FbgZ2qulZVM4rGO9N2AW87i00GjqpqV2Am8Kfz20Xji9sv6Uy3lo148L1kTuUWuF2OMcZlvhw59AdSVDVVVXOBRcCIUuabATwFZJexnnHAwrNHikg3oCXwhTNqBDDfef8WMERExIc6zXkICwlixsg40o6e4p+fWcd8xtR1voRDW2C313CaM66YiPQB2qvq0nLWM4ZSwgFPaLyhqkUd/RRvT1XzgeNAi7MXEpEpIpIoIonp6ek+7IapyMDOLbjh4nbM/l8qKQetYz5j6jJfwqG0X+3FPbaJSBCe0z/3lLkCkQFAlqpuKGXyWM4MjXK3VzxCdbaqJqhqQlRUVFmbNpX0hx/H0CAshAfe3cDpvDbG1DW+hEMa0N5ruB2w12s4AogDlovITmAgsKSoUdpxdgAAICIXASGqmlTa9kQkBGgC2C28NaRFo3B+PzSGlalHeOfbPW6XY4xxiS/hsAboJiKdRCQMzxf9kqKJqnpcVSNVNVpVo4GVwHBVTYTiI4sb8bRVnK20doglwATn/WjgU7WfsDVqbL/29OnQlMf+vYnjWXlul2OMcUGF4eCc958KLAM2AYtVNVlEpovIcB+2cSmQpqqppUy7iZLhMA9oISIpwN1AmZfOmuoRFCQ8NvJCjp3K40/LrGM+Y+oiqQ0/yhMSEjQxMdHtMmqdGUs38uJXO3j7rsH06dDM7XKMMVVMRJJUNaG0aXaHtCnTr6/qTqsIT8d8+dYxnzF1ioWDKVOj8BAevC6WjftOMH/FLrfLMcbUIAsHU65hca25vEcUT/93C/uPl3V/ozGmtrFwMOUSEaYPjyO/UJmxdKPb5RhjaoiFg6lQhxYNmHpFV/793T6WbznodjnGmBpg4WB8MuWyznSOasiD7yVz/JTd+2BMbWfhYHwSHhLME6MuZN/xU4yfu5KjJ3PdLskYU40sHIzPBnRuwexbE9h6IJNxc1ZyKDPH7ZKMMdXEwsFUyhUxLXlxQj92Hj7J2NkrOXjCrmAypjaycDCVdkm3SOZP6s++Y6e4adYKe/a0MbWQhYM5JwM6t+CVyQM4nJnLTbNWsPtIltslGWOqkIWDOWd9OzZjwR0DyMjOZ8ysFew4dNLtkowxVcTCwZyX3u2asvCOgWTnFzJm1gp7gpw5L6dyCygsDPzOQGsD65XVVIltBzK4ee4qCguV124fQM82jd0uyQSYl7/awYx/b6JeSBA9WkcQ06YxMa0jiGndmB6tI2hSP9TtEmud8npltXAwVSY1PZOb56wiO7+A1yYPIK5tE7dLMgGgsFB57INNzPtyB5f3iCK6RUM27TvB5v0ZZ9xw2bZpfU9oOMHRs3UEnSIbEhJsJ0DOlYWDqTHfH85i3JyVnMjO45Xb+ttzIEy5svMK+NWitXyYvJ+Jg6P547WxBAd5HiOvquw/kc3m/Rls3pfB5v0n2Lwvg+3pmeQ7p57CgoPo2rIRMW0i6Nm6MTFtIujROoKoRuGIlPY4euPNwsHUqLSjWYyfu4rDmbm8NKkf/aKbu12S8UOHM3O4/ZVE1u4+xgPXxDL5kk4+LZebX8j29MzisNi83xMcB06cvimzRcMwYtqcPiXVs3VjurVqRL3Q4OranYBk4WBq3P7j2dw8dyX7jmUzb0ICg7tGul2S8SOp6ZlMenkN+49n88zYeIbGtTnvdR45mcvm/SfY4nWkseVABtl5ngdVBQl0imxITOvGxaemYlpH0K5Z/Tp7lHHe4SAiQ4FngGBgrqo+WcZ8o4E3gX6qmigi44Hfes3SG7hYVdeKSBjwT+ByoBC4X1X/JSITgT8De5xl/qmqc8urz8LBP6Vn5HDL3FXsPHyS2T9J4LLuUW6XZPxA4s4j3P5KIkEizJ2QwMXVeOqxoFDZdfgkW/ZnsGl/Bpudtozvve7LaRQeUqIto3vrCBrXq/0N4OcVDiISDGwFrgLSgDXAOFXdeNZ8EcC/gTBgqqomnjX9QuA9Ve3sDD8CBKvqAyISBDRX1UNOOCSo6lRfd9DCwX8dOZnLLXNXkXIwk+fGX8yVsa3cLsm4aOn6vdy9eB1tm9bn5Un96NiioSt1ZObks/XAmW0Zm/ef4ER2fvE8bZvWp6f3qak2EUS3qF0N4OWFQ4gPy/cHUlQ11VnZImAEcPaTX2YATwG/KWM944CFXsO3ATEAqloIHPKhFhNgmjcMY+EdA/nJi6v46WtJ/GNcH4ZdeP6nEExgUVVm/S+VJ/+zmYSOzZjzkwSaNQxzrZ5G4SFc3KHZGUctqsq+49ls3n+CTfsyPKen9p/gsy3pFBQ1gIcE0b1VI3q0alwcHDFtIohsFO7WrlQbX8KhLbDbazgNGOA9g4j0Adqr6lIRKSscxuAJFUSkqTNuhohcDmzHc7RxwBl/g4hciueI5dequvvslYnIFGAKQIcOHXzYDeOWJg1Cee32AUx6aQ1TF37L0wWFjIhv63ZZpobkFxTy0JJkFqz6nmt6t+GvN17klw3DIsIFTetzQdP6/DDm9BFuTn4BKQcznbDIYNO+E/xvWzr/+iateJ7IRuHOPRmn2zK6tgzsBnBfwqG0lpric1HOKaGZwMQyVyAyAMhS1Q1e220HfKWqd4vI3cBfgFuB94GFqpojIj8F5gM/LFGA6mxgNnhOK/mwH8ZFEfVCmX9bfybPX8Ov3lhLTn4hNyW0d7ssU81O5uTzi4Xf8unmg9x5WWd+f3UMQUGB1fgbHhJMrwua0OuCM+/bOZyZU6It49WVu8jJ9zSABweJ0wAeQc82jenRKoKYNhG0bRoYDeC+tDkMAh5W1aud4fsAVPUJZ7gJnl/+mc4irYEjwPCidgcRmQmkq+rjzrA480eoaqGItAc+VNVeZ207GDiiquXeTWVtDoHjVG4BU15N5Itth3hsVBzjB3R0uyRTTQ6eyOa2+WvYuPcE00fEccvA2v/fuqBQ2Xn45Om2DOfU1O4jp3sujggPKb4fI6a15/RU91YRRLjQAH6+bQ5rgG4i0gnPFURjgZuLJqrqcaD4OkURWQ78xisYgoAbgUu9llEReR/PlUqfAkNw2jBEpI2q7nNmHQ5s8mkvTUCoHxbMnJ8k8PMF33D/OxvIySvkNh+vbzeBY+uBDCa9tIajWbnMm9CPK2Jaul1SjQgOErpENaJLVCOu6X26bS0jO8/TAO51me173+7ltZzvi+dp37x+ibaM6BYNi28KrGkVhoOq5ovIVGAZnktZX1TVZBGZDiSq6pIKVnEpkFbUoO3l98CrIvI3IB2Y5IyfJiLDgXw8RyATfd4bExDqhQbz/C19mbbwW6Yv3UhuQSE/vayL22WZKvJ1yiHufC2JeqHBLL5zkHWjgue0at+Ozenb8fQNoarKnmOnzmjL2Lw/g8+2HCxuAA8PCaJ7qzPbMmJaR9CiBhrA7SY445q8gkLuXryO99ft5e6rujNtSDe3SzLn6e1v0vj9v9bTKbIhL03qT9um9d0uKeBk53kawDc7bRlbDmSwaV/GGY/ljYo43QB+be8LuKh903LWWLbzPa1kTLUIDQ7ib2PiCQsO4umPtpKbX8g9P+oeEI115kyqyj8+TeHpj7YyuEsLnr+lr/Wieo7qhQYT17ZJiSOu9Iyc4stri9oy5q/YRY/Wjc85HMpj4WBcFRwk/Hl0b8JChH9+lkJOfgF/+HFPC4gAkldQyB/e/o43k9K4/uK2PHl9b8JCas+NYv4iKiKcqIhwLul2uiua/IJCCqrp7I+Fg3FdUJDw+KgLCQ8JZs4XO8jJL+Th63oF3CWPddGJ7Dx+9to3fJlyiF8O6cavruxmwV6DQoKDqu1L3MLB+AUR4aHrYgkLCWL2/1LJzS/k8VEXWkD4sb3HTnHby2tIOZjJn0f35ka7b6VWsXAwfkNEuG9YDOEhQfzj0xRy8wt5anTvWtWXTW2RvPc4t728hqycAl6e1P+MUx2mdrBwMH5FRLjnRz0ICw7irx9tJbegkJlj4gm1gPAby7cc5OcLvqFx/VDevGsQMa3tkbC1kYWD8Uu/GNKN8NAgHv9gM7n5hfzj5j6EhwRuPzW1xcLV3/PAuxvo0SqClyb1o1Xjem6XZKqJ/RwzfmvKpV14+LpY/rvxAD99NYnsvAK3S6qzCguVpz7czH1vf8clXSNZ/NNBFgy1nIWD8WsT/68Tj4+6kOVb07l9fiKnci0galpOfgG/emMtzy3fzrj+HZg3IYFG4XbSobazcDB+7+YBHfjz6Iv4evshJr60mpM5+RUvZKrEsaxcbp23miXr9vK7oT14fFScXSBQR9h/ZRMQRvdtx8wx8STuOspPXlzNiew8t0uq9b4/nMX1z3/N2u+P8czYeH52eVe7h6EOsXAwAWNEfFuevbkP69OOccvcVRzLynW7pFpr3e5jXP/8VxzOzOXVyf3t4Ux1kIWDCShD49rwwi192bwvg5vnrOKwV2dkpmr8N3k/Y2avoH5YMP+6azADOrdwuyTjAgsHE3CG9GzF3AkJbE/PZNyclRzMyHa7pFrj5a92cOdrSfRo3Zi37/o/urZs5HZJxiUWDiYgXdo9ipcn9Sft6CnGzlrJ/uMWEOejsFCZsXQjD7+/kSt7tmLRHQOJiqj+ZwYY/2XhYALWoC4teOW2/hzMyOGmWStIO5rldkkBKXsX9xwAAA/OSURBVDuvgJ8t+IZ5X+5g4uBoXrilL/XD7IbDus7CwQS0hOjmvHb7AI5l5TJm1kp2HT7pdkkB5XBmDuPmrGTZxv08eG0sDw/v5dpjKY1/sXAwAS++fVNev2MgWbn53DRrBdvTM90uKSCkpmdy/fNfs3HvCZ4f39ee5W3O4FM4iMhQEdkiIikicm85840WERWRBGd4vIis9XoViki8My1MRGaLyFYR2SwiNzjjw0XkDWdbq0Qk+vx309R2cW2bsHDKQAoKlTGzVrJlf4bbJfm1xJ1HuP75r8nMzmfRlIEMjWvtdknGz1QYDiISDDwLDANigXEiElvKfBHANGBV0ThVXaCq8aoaD9wK7FTVtc7k+4GDqtrdWe/nzvjJwFFV7QrMBP50rjtn6paY1o1ZNGUQQQJjZ68gee9xt0vyS0vX7+Xmuato1iCMt382mD4dmrldkvFDvhw59AdSVDVVVXOBRcCIUuabATwFlHXZyDhgodfwbcATAKpaqKqHnPEjgPnO+7eAIWK3ZRofdW3ZiMV3DqJ+aDA3z1nFut3H3C7Jb6gqL3y+namvf0vvtk14+67BdGzR0O2yjJ/yJRzaAru9htOcccVEpA/QXlWXlrOeMTjhICJFT8OeISLfiMibItLq7O2paj5wHChxF46ITBGRRBFJTE9P92E3TF0RHdmQN+4cROP6IdwydxVJu464XZLr8gsKeeDdDTz5n81c07sNr90+gGYNw9wuy/gxX8KhtF/txU+0FpEgPKd/7ilzBSIDgCxV3eCMCgHaAV+p6sXACuAvvmyveITqbFVNUNWEqKgoH3bD1CXtmzdg8Z2DiIwI59Z5q1mZetjtklxzMiefKa8msWDV99x5WWf+MbYP9ULtUlVTPl/CIQ3wfjhsO2Cv13AEEAcsF5GdwEBgSVGjtGMsZ55SOgxkAe84w28CF5+9PREJAZoA9tPPVFqbJvV5Y8pA2jatz8SXVvPFtrp3hHnwRDZjZq9g+ZaDPDoyjvuG9bTnchuf+BIOa4BuItJJRMLwfNEvKZqoqsdVNVJVo1U1GlgJDFfVRCg+srgRT1tF0TIKvA9c7owaAmx03i8BJjjvRwOfOvMbU2ktG9dj0ZSBdIpsxOT5iXy6+YDbJdWYrQcyGPXc16Smn2TehH7cMrCj2yWZAFJhODjn/acCy4BNwGJVTRaR6SIy3IdtXAqkqWrqWeN/DzwsIuvxXMlUdFpqHtBCRFKAu4EyL501xhctGoWz8I4B9GgVwZ2vJvHhhv1ul1Ttvt5+iBue/5rcgkIW3zmIK2Jaul2SCTBSG36UJyQkaGJiottlGD93IjuPiS+uZl3acf42Jp7rLrrA7ZKqxdvfpPH7f62nU2RDXprUn7ZN67tdkvFTIpKkqgmlTbM7pE2d0bheKK9MHkDfjs345aJv+VdSmtslVSlV5e+fbOPuxevoF92cN3862ILBnDMLB1OnNAoP4eVJ/RjUpQW/eWsdC1d/73ZJVSKvoJDfvbWepz/ayvUXt+XlSf1pUj/U7bJMALNwMHVOg7AQ5k3ox2Xdo7jv7e94ZcVOt0s6Lyey85j00hreTErjl0O68dcbLyIsxP7XNufH/oJMnVQvNJhZt/blqthWPPheMnO/OPt6icCw99gpbnphBStTD/Pn0b359VXd7TnPpkpYOJg6KzwkmOfGX8w1F7bh0X9v4tnPUtwuqVKS9x5n1HNfsefoKV6e1J8bE9pXvJAxPgpxuwBj3BQaHMQzY+MJCwniz8u2kJNXEBC/vpdvOcjPF3xDk/qhvHnXIGJaN3a7JFPLWDiYOi8kOIi/3HgRYcFB/P3TFHIKCrl3aIzfBsTC1d/zwLsb6NEqgpcm9aNV43pul2RqIQsHY4DgIOGJ6y8kLCSIWZ+nkpNXyEPXxfpVQBQWKn/57xaeW76dy7pH8ez4i2kUbv8Lm+phf1nGOIKChOkjehEWEsS8L3eQW1DIoyPi/KIvopz8An775nqWrNvLuP4dmDGiFyHB1mRoqo+FgzFeRIQHrulJeEgQzy3fTm5+IX+6oberz1U+lpXLlFeTWL3jCL8b2oO7LuviV0c0pnaycDDmLCLCb6/uQXhIMDM/3kpufiFP33SRK7/Udx/JYsJLq0k7copnxsYzIr5txQsZUwUsHIwphYjwyyu7ERoiPPXhFvIKCnlmbJ8avbls3e5jTJ6/hrwC5dXJ/RnQucQzr4ypNhYOxpTjZ5d3JTwkmBlLN5K3IIlnx19MeEj1Pyjnv8n7mbboW6Iiwlk0sT9dWzaq9m0a481atIypwORLOjFjZBwfbzrIHa8kkZ1XUK3be/mrHdz5WhI9Wjfm7bv+z4LBuMLCwRgf3DqwI0/d0JsvtqUz6aU1ZOXmV/k2CguVGUs38vD7G7myZysW3TGQqIjwKt+OMb6wcDDGRzf1a8/Mm+JZteMwE15cTUZ2XpWtOzuvgJ8t+IZ5X+5g4uBoXrilL/XD7DnPxj0WDsZUwsg+bfnHuIv59vtj3DJvNcezzj8gDmfmMG7OSpZt3M+D18by8PBerl46awxYOBhTadf0bsPzt/Rl094T3Dx3JUdP5p7zulLTM7n++a/ZuPcEz4/vy22XdKrCSo05dz6Fg4gMFZEtIpIiImU+01lERouIikiCMzxeRNZ6vQpFJN6ZttxZZ9G0ls74iSKS7jX+9qrYUWOq0lWxrZj9k76kHMxk3JyVpGfkVHodiTuPcP3zX5OZnc+iKQMZGte6Gio15txUGA4iEgw8CwwDYoFxIhJbynwRwDRgVdE4VV2gqvGqGg/cCuxU1bVei40vmq6qB73Gv+E1fu657Zox1evyHi15cWI/dh3OYuzsFRw4ke3zskvX7+Xmuato3iCMt382mD4dmlVjpcZUni9HDv2BFFVNVdVcYBEwopT5ZgBPAWX9HzIOWHhOVRrjp/6vayTzb+vP/uPZ3DRrBXuOnSp3flXlhc+3M/X1b7moXRP+dddgOrZoWEPVGuM7X8KhLbDbazjNGVdMRPoA7VV1aTnrGUPJcHjJOXX0Rzmzs5gbRGS9iLwlIqU+wUREpohIoogkpqen+7AbxlSP/p2a8+rtAzhyMpcxs1aw+0hWqfPlFxTywLsbePI/m7m2dxtenTyAZg3DarhaY3zjSziUdtmEFk8UCQJmAveUuQKRAUCWqm7wGj1eVS8EfuC8bnXGvw9Eq2pv4GNgfmnrVNXZqpqgqglRUVE+7IYx1efiDs14/faBZGTnc9OsFew4dPKM6Sdz8pnyahILVn3PTy/rwt/H9qFeqF2qavyXL+GQBnj/em8H7PUajgDigOUishMYCCwpapR2jOWsowZV3eP8mwG8juf0Fap6WFWLWvfmAH193Rlj3HRhuyYsvGMgufmF3DRrBdsOZABw8EQ2Y2avYPmWgzw6Mo57h8X4RTfgxpTHl3BYA3QTkU4iEobni35J0URVPa6qkaoararRwEpguKomQvGRxY142ipwxoWISKTzPhS4FtjgDLfx2vZwYNN57J8xNSr2gsYsmjIQgLGzV7J0/V5GPfc1qeknmTehH7cM7Ohyhcb4psJwUNV8YCqwDM8X9WJVTRaR6SIy3IdtXAqkqWqq17hwYJmIrAfWAnvwHCUATBORZBFZh+fqp4k+740xfqBbqwgW3zmIsJAgpr7+LbkFhSy+cxBXxLR0uzRjfCaqWvFcfi4hIUETExPdLsOYM+w+ksW8L3dw+w860a5ZA7fLMaYEEUlS1YTSplmX3cZUk/bNG/Dw8F5ul2HMObHuM4wxxpRg4WCMMaYECwdjjDElWDgYY4wpwcLBGGNMCRYOxhhjSrBwMMYYU4KFgzHGmBJqxR3SIpIO7DrHxSOBQ1VYTlWxuirH6qo8f63N6qqc86mro6qW2q11rQiH8yEiiWXdPu4mq6tyrK7K89farK7Kqa667LSSMcaYEiwcjDHGlGDhALPdLqAMVlflWF2V56+1WV2VUy111fk2B2OMMSXZkYMxxpgSLByMMcaUUKvDQUSGisgWEUkRkXtLmT5RRNJFZK3zut1r2gQR2ea8JvhRXQVe45ecvWx11uXMc5OIbHQe5fq613jXPq8K6nLt8xKRmV7b3ioix7ymufn3VV5dbn5eHUTkMxH5VkTWi8iPvabd5yy3RUSu9oe6RCRaRE55fV4v1HBdHUXkE6em5SLSzmva+f99qWqtfAHBwHagMxAGrANiz5pnIvDPUpZtDqQ6/zZz3jdzuy5nWqaLn1c34NuizwJo6SefV6l1uf15nTX/L4AX/eHzKqsutz8vPA2rdznvY4GdXu/X4Xn2fCdnPcF+UFc0sMHFz+tNYILz/ofAq1X591Wbjxz6AymqmqqqucAiYISPy14NfKSqR1T1KPARMNQP6qpOvtR1B/Cs85mgqged8W5/XmXVVZ0q+99xHLDQee/251VWXdXJl7oUaOy8bwLsdd6PABapao6q7gBSnPW5XVd18qWuWOAT5/1nXtOr5O+rNodDW2C313CaM+5sNziHZW+JSPtKLlvTdQHUE5FEEVkpIiOrqCZf6+oOdBeRr5ztD63Esm7UBe5+XoDn8B/PL95PK7tsDdcF7n5eDwO3iEga8AGeoxpfl3WjLoBOzummz0XkB1VUk691rQNucN6PAiJEpIWPy1aoNoeDlDLu7Ot23weiVbU38DEwvxLLulEXQAf13Cp/M/A3EelSg3WF4DmFczmeX5xzRaSpj8u6URe4+3kVGQu8paoF57BsZZ1PXeDu5zUOeFlV2wE/Bl4VkSAfl3Wjrn14Pq8+wN3A6yLSmKrhS12/AS4TkW+By4A9QL6Py1aoNodDGuD9i7sdZx0OquphVc1xBucAfX1d1qW6UNW9zr+pwHKgT03V5czznqrmOYf3W/B8Kbv6eZVTl9ufV5GxnHnqxu3Pq6y63P68JgOLne2vAOrh6VTO7c+r1Lqc01yHnfFJeNoIutdUXaq6V1Wvd8LpfmfccR/3qWLV0ZjiDy88vyZT8Rw2FzXo9DprnjZe70cBK/V0g84OPI05zZz3zf2grmZAuPM+EthGOY2N1VDXUGC+1/Z3Ay384PMqqy5XPy9nvh7ATpwbTv3h76ucutz++/oPMNF53xPPF5oAvTizQTqVqmuQPp+6oorqwNNwvKeG/+4jgSDn/WPA9Kr8+zrvnfDnF55DwK14Ev1+Z9x0YLjz/gkg2fngPwNivJa9DU/DVwowyR/qAgYD3znjvwMm13BdAjwNbHS2P9ZPPq9S63L783KGHwaeLGVZ1z6vsupy+/PC08D6lbP9tcCPvJa931luCzDMH+rCc76/6P/Tb4Drariu0XgCfCswFyfYq+rvy7rPMMYYU0JtbnMwxhhzjiwcjDHGlGDhYIwxpgQLB2OMMSVYOBhjjCnBwsEYY0wJFg7GGGNK+H/5RfDEqTbVCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "#train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "#train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = subsample_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.47907025, -0.4767751 , -0.4754073 , -0.47663534, -0.47638331])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  8.5min remaining:  2.1min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 10.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=1.0,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=30,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.8,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4,\n",
       "             param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4739258649144454\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXwV9b3/8dcnYcew72tQQGQRkBCwrdRqbbFWsEUri8gm2t6q91a9vd5arRVt1Xpra/VnK4IiIm6tEqnVuhSvtQIJAhJAMKwJIAQiawiQ5PP7I5PcQxLICYTMSfJ+Ph7nkVm+M/OZ4XDe5ztzzhxzd0RERCLFhV2AiIjEHoWDiIiUoXAQEZEyFA4iIlKGwkFERMqoF3YBVaFNmzaemJgYdhkiIjXKsmXLdrt72/Lm1YpwSExMJC0tLewyRERqFDPbcqJ5Oq0kIiJlKBxERKQMhYOIiJShcBARkTIUDiIiUobCQUREylA4iIhIGQoHEZEa6OCRfB7822dk5uSekfXXii/BiYjUFe7OX1ft4P6Fa/lifx6dWzZm4vDuVb4dhYOISA2Rsesgv0hJ56OMPfTr1IwnJlzAkO4tz8i2FA4iIjEu92g+f3g/g6c/3Eij+vHcN7ofE4Z1Jz7Oztg2FQ4iIjHK3Xkr/QtmLFzD9n15jLmgC3de3oe2CQ3P+LYVDiIiMWjT7kP8ImU1/7s+mz4dEvj9uMEMTWxVbdtXOIiIxJDDRwt44h8ZPPW/G2lYL45fXNmXicO7Uy++ej9cqnAQEYkB7s47a3byyzfWsG3vYb43uDP/fXkf2jVrFEo9CgcRkZBt2XOIe1NW84912fRufxYv3TicYWe3DrUmhYOISEjyjhXw5KINPPnBBurHGT+/4jwmfSWR+tV8Cqk8CgcRkRC8t3Yn976xmsycw4wa2Im7rjiP9iGdQiqPwkFEpBpl5uTyyzdW8+7aXfRsdxYv3DCMr/RsE3ZZZUTVdzGzkWa2zswyzOzOk7S72szczJKC8QlmtiLiUWhmg8wsodT03Wb2u2CZhmb2UrCtJWaWWBU7KiISprxjBTz23ud887cf8K8Ne7jz8j68eetFMRkMEEXPwczigSeAy4AsINXMUtx9Tal2CcCtwJLiae4+D5gXzB8ALHD3FcHsQRHLLgP+EoxOA750955mNhZ4CLj21HZPRCR8i9bt4t6U1Wzek8sVAzpy1xXn0alF47DLOqloTislAxnuvhHAzF4ERgNrSrWbATwM3HGC9YwD5peeaGa9gHbAh8Gk0cC9wfCrwONmZu7uUdQqIhIzsr7MZcbCNby9eidnt2nK3GnJXNSrbdhlRSWacOgMZEaMZwHDIhuY2WCgq7svNLMThcO1FL3wlzYOeCnixb9ke+6eb2b7gNbA7lLbvBG4EaBbt25R7IaISPU4kl/A0x9u4g/vfw7Af377XG64qAcN68WHXFn0ogmH8u7sVPIu3szigEeBySdcgdkwINfd08uZPRaYGO32Sia4PwU8BZCUlKRehYjEhA8/z+YXC1azcfchvt2vPfdc2Y/OMX4KqTzRhEMW0DVivAuwPWI8AegPLDIzgA5AipmNcve0oM1Yyj+lNBCo5+7LytlelpnVA5oDOdHtjohIOHbsO8z9C9fy11U76N66Cc9MGco3zm0XdlmnLJpwSAV6mVkPYBtFL/Tji2e6+z6g5HK7mS0C7igOhqBncQ0wopx1l3cdIgWYBHwMXA28r+sNIhKrjuYXMvujTTz23ucUFDq3X9ab6SPOplH9mnMKqTwVhkNw3v9m4G0gHpjt7qvN7D4gzd1TKljFCCCr+IJ2KT8AvlNq2ixgrpllUNRjGFtRjSIiYfhXxm7uXpDOhuxDfPO89vziyr50bdUk7LKqhNWGN+VJSUmelpZWcUMRkSrwxb48HnhzLW+s3E7XVo2598p+XHpe+7DLqjQzW+buSeXN0zekRUSidKygkDn/2syj76znWKHz75f24kcXn1PjTyGVR+EgIhKFxRv3cM+CdNbvPMg3zm3LvaP60b1107DLOmMUDiIiJ7Frfx6/enMtr6/YTucWjXlq4hAu69ue4NOZtZbCQUSkHPkFhTz38RYefWc9R/ILueWSnvzbxT1p3KD2nUIqj8JBRKSU1M053P16Op99cYARvdvyy1H96NGm9p5CKo/CQUQkkH3gCA/+7TP+/EkWnZo34o/XXcC3+3Wo9aeQyqNwEJE6L7+gkHlLtvLI39eRd6yAH118Drdc0pMmDeruS2Td3XMREWDZli+5+/V01uzYz9d6tuHeUf3o2e6ssMsKncJBROqkPQeP8NBbn/FyWhbtmzXk8fGDuWJAxzp5Cqk8CgcRqVMKCp35S7fym7fXcehIPjeNOJtbLu3FWQ31chhJR0NE6owVmXu5+/V0Vm3bx/CzWzFjdH96tU8Iu6yYpHAQkVrvy0NHefjtz3gxNZO2ZzXk92MHMWpgJ51COgmFg4jUWoWFzktpmTz01mccyMtn2ld78O/f7EVCo/phlxbzFA4iUiutytrHzxekszJzL8mJrbjvqn706dAs7LJqDIWDiNQqe3OP8sjf1zFvyVZaN23Io9cO5KpBnXUKqZIUDiJnSPq2ffz6b2tp2qAe3Vo1oVvrJkV/WzWhc8vGNerH5muCwkLn1WVZPPjWZ+zNPcqkCxP5yWW9ad5Yp5BOhcJB5AzYtPsQk2YvxQxaNmnAB+uzOZJfWDLfDDo2a0TXVk3oHoRG11b/Fx6tmjbQO91KSN+2j3sWpPPJ1r0M6d6SGaOH0beTTiGdDoWDSBXbuT+PibOW4MDLN13IOW3Pwt3JPnCErTm5bNmTy9acXDJziv4uWpfNrgNHjlvHWQ3rBWHRuCQwurVuWtTraNGYBvXiwtm5GLPv8DF++/d1zF28hZZNGvCbq89nzAVdiItTsJ4uhYNIFdp3+BiTZi8l59BR5k8fzjlti27DYGa0a9aIds0akZTYqsxyh48WkPllLluD4Ch+bMg+xKJ1x/c64gw6Nm9M11aN6d6qKd1aH9/raNmkfq3vdbg7f/lkG7/+21pyDh3luuHduf2yc2neRKeQqorCQaSK5B0rYPqcNDZkH2T25KEM7Noi6mUbN4ind/sEepfzhazCQif7YPm9jvfX7SK7VK8joaTX0aQkOLoH451qQa9j7Y793LMgndTNXzKoawuenZJM/87Nwy6r1lE4iFSB/IJCbpm/nNQtOTw2djAX9WpbZeuOizPaN2tE+2aNGFpOryP3aD6ZOYdLehuZObls2XOIz3cd4P11uzhaTq+jW3CtI7LH0a1VE1rEcK9jf94xfvfO58z5eDPNGtXjoTEDuGZIV51COkOiCgczGwn8HogHnnb3B0/Q7mrgFWCou6eZ2QTgPyOanA9c4O4rzKwB8DhwMVAI3OXufzazycBvgG3BMo+7+9OV3jORauLu/Oy1VbyzZie/HNWPKwd2qtbtN2lQj3M7JHBuh/J7HbtKrnUcKulxbM3J5d21u9h9sFSvo1G948KiODy6ty7qddSPr/5eh7uzYMV2HnhzLbsPHmFccjd++u1zadGkQbXXUpdUGA5mFg88AVwGZAGpZpbi7mtKtUsAbgWWFE9z93nAvGD+AGCBu68IZt8F7HL33mYWB0S+JXrJ3W8+9d0SqT6/eXsdL6dlceslPZn0lcSwyzlOXJzRoXkjOjRvRHKPsr2OQ0fyj7vWkZmTy5acXNbtPMB7a3dxtOD4XkenFpEXyEv3Oqr+xXr9zgPc/Xo6SzblcH6X5jx9fVKlTtfJqYum55AMZLj7RgAzexEYDawp1W4G8DBwxwnWMw6YHzE+FegD4O6FwO7oyxaJDbP+uYn/t2gD45K78ZPLeoddTqU1bViPPh2alfvN4cJCZ+eBPLbuKQqM43sdO9l98Ohx7Zs1qlcSGJGnq7q3akrHFo0q1es4eCSf37+7nmc+2kzThvV44Hv9GTu0G/E6hVRtogmHzkBmxHgWMCyygZkNBrq6+0IzO1E4XEtRqGBmxdE/w8wuBjYAN7v7zmD6GDMbAawHfuLumaVXZmY3AjcCdOvWLYrdEKlary/fxoyFaxjZrwP3X9U/Zs/Vn6q4OKNj88Z0bN6YYWe3LjP/4JH8ksAo/rtlTy6f7TjAO2t2cqzAS9rGxxmdWjSK6Gk0Pa7XUfwpI3dn4ac7uP+va9i5/whjh3blpyP70KqpTiFVt2jCobxnfMm/enBK6FFg8glXYDYMyHX39IjtdgE+cvfbzOw24BFgIvAGMN/dj5jZD4E5wCVlCnB/CngKICkpyUvPFzmTFq3bxR2vrGT42a343dhBdfId7VkN63Fex2ac17Fsr6Og0Nm5P6+op1Hq47l/X72TPYfK9jq6t24KwKpt++jfuRlPXjeEC7q1rJZ9kbKiCYcsoGvEeBdge8R4AtAfWBS8c+oApJjZKHdPC9qM5fhTSnuAXOC1YPwVYBqAu++JaDcTeCiqPRGpJsu3fsmPnv+E3u0TmHl9Eo3q6zYYpRX1FBrTqUVjhpfT6ziQd6zkE1YlvY6cXPYcPMKM0f0YP6x7nQzcWBJNOKQCvcysB0WfIBoLjC+e6e77gDbF42a2CLijOBiCnsU1wIiIZdzM3qDok0rvA5cSXMMws47uviNoOgpYe4r7JlLlMnYdYMqzqbRr1pA5U5N16+dTlNCoPn071dctLmJYheHg7vlmdjPwNkUfZZ3t7qvN7D4gzd1TKljFCCCr+IJ2hP8C5prZ74BsYEow/VYzGwXkAzmc5HSVSHXavvcwE2ctpV5cHHOnDqNtQsOwSxI5Y8y95p+uT0pK8rS0tIobipyiLw8d5Zo/fczOfXm8eNNw+nXSN3Kl5jOzZe6eVN48fUNapAK5R/OZOieVrTm5PDc1WcEgdULNvsmKyBl2rKCQf5v3CSsz9/LY2MHlXlwVqY3UcxA5gcJC56evfsqiddn8+vsDGNm/Q9gliVQb9RxEyuHuPPDmWl5bvo07vtWbccn6oqXULQoHkXL88YONzPrnJiZ/JZEff6Nn2OWIVDuFg0gpL6dm8tBbnzFqYCfu+W7fWndbDJFoKBxEIryzZid3/uVTLurVhkeuGajfCpA6S+EgEli6KYebX/iEAV1a8MfrhtT4X0wTOR169otQ9NOT0+ak0rllY56ZPJSmDfVBPqnbFA5S52Xm5DJp9lKaNqjHc1OTdXtoEfQ9B6njdh88wvWzl3Ikv5BXfnghXVo2CbskkZignoPUWQeP5DPlmVR27DvM7MlJ9G5f9jeYReoq9RykTjqSX8BNc9NYs2M/M68fwpDuZX9fWaQuU89B6pyCQue2l1byUcYeHh5zPpf0aR92SSIxR+EgdYq7c2/Kav66agd3fec8xgzpEnZJIjFJ4SB1ymPvZTB38RZuGnE200ecHXY5IjFL4SB1xvOLt/Dou+sZc0EX7ry8T9jliMQ0hYPUCW+u2sHdC9K5tE87HhwzQPdLEqmAwkFqvX9l7OY/XlzBkG4teXz8BdSP19NepCL6XyK1Wvq2fUx/Lo0ebZoya9JQGjeID7skkRpB4SC11qbdh5g0eyktmjRgztRkmjepH3ZJIjVGVOFgZiPNbJ2ZZZjZnSdpd7WZuZklBeMTzGxFxKPQzAYF8xqY2VNmtt7MPjOzMcH0hmb2UrCtJWaWePq7KXXNrv15TJy1BAeem5ZMh+aNwi5JpEapMBzMLB54Argc6AuMM7O+5bRLAG4FlhRPc/d57j7I3QcBE4HN7r4imH0XsMvdewfr/SCYPg340t17Ao8CD53qzkndtO/wMa6fvZScQ0d5ZvJQzml7VtglidQ40fQckoEMd9/o7keBF4HR5bSbATwM5J1gPeOA+RHjU4FfA7h7obvvDqaPBuYEw68Cl5o+WiJRyjtWwPQ5aWzIPsifJg5hYNcWYZckUiNFEw6dgcyI8axgWgkzGwx0dfeFJ1nPtQThYGbF/2NnmNknZvaKmRXfw6Bke+6eD+wDWpdemZndaGZpZpaWnZ0dxW5IbZdfUMgt85eTuiWH3/5gEBf1aht2SSI1VjThUN67di+ZaRZH0emf20+4ArNhQK67pweT6gFdgI/c/QLgY+CRaLZXMsH9KXdPcvektm31IlDXuTt3vZbOO2t2cu+V/bhyYKewSxKp0aIJhyyga8R4F2B7xHgC0B9YZGabgeFASvFF6cBYjj+ltAfIBV4Lxl8BLii9PTOrBzQHcqKoU+qw37y9jpfSMrnlkp5M+kpi2OWI1HjRhEMq0MvMephZA4pe6FOKZ7r7Pndv4+6J7p4ILAZGuXsalPQsrqHoWkXxMg68AVwcTLoUWBMMpwCTguGrgfeD9iLlmvXPTfy/RRsYl9yN2y7rHXY5IrVChb/n4O75ZnYz8DYQD8x299Vmdh+Q5u4pJ18DI4Asd99Yavp/AXPN7HdANjAlmD4rmJ5BUY9hbPS7I3XN68u3MWPhGkb268D9V/XXbTFEqojVhjflSUlJnpaWFnYZUs0WrdvFDXPSSEpsybNTkmlUX99+FqkMM1vm7knlzdM3pKVGWr71S370/Cf0bp/AzOuTFAwiVUzhIDVOxq4DTHk2lbYJDXl26lASGum2GCJVTeEgNcr2vYe5ftZS6sXFMXdaMu0SdFsMkTNB4SA1xpeHjnL97KUcyMtnztShdG/dNOySRGqtCj+tJBILco/mM3VOKltzcpkzJZl+nZqHXZJIraaeg8S8YwWF/Nu8T1iZuZfHxg7mwnPK3E1FRKqYeg4S0woLnZ+++imL1mXz6+8PYGT/DmGXJFInqOcgMcvdeeDNtby2fBt3fKs345K7hV2SSJ2hcJCY9ccPNjLrn5uY/JVEfvyNnmGXI1KnKBwkJr2clslDb33GqIGduOe7fXVbDJFqpnCQmPPOmp38919WcVGvNjxyzUDi4hQMItVN4SAxZemmHG5+4RP6d2rGH68bQoN6eoqKhEH/8yRmrN2xn2lzUuncsjGzJw+laUN9mE4kLAoHiQmZOblMmr2Upg3q8dzUZFqf1TDskkTqNL01k9DtPniE62cv5Uh+Ia/88EK6tGwSdkkidZ56DhKqg0fymfJMKjv2HWb25CR6t08IuyQRQT0HCdGR/AJumpvGmh37mXn9EIZ0bxV2SSISUM9BQlFQ6Nz20ko+ytjDw2PO55I+7cMuSUQiKByk2rk796as5q+rdvCz7/RhzJAuYZckIqUoHKTaPfZeBnMXb+GmEWdz44hzwi5HRMqhcJBq9fziLTz67nrGXNCFOy/vE3Y5InICUYWDmY00s3VmlmFmd56k3dVm5maWFIxPMLMVEY9CMxsUzFsUrLN4Xrtg+mQzy46YfkNV7KiE781VO7h7QTqX9GnHg2MG6H5JIjGswk8rmVk88ARwGZAFpJpZiruvKdUuAbgVWFI8zd3nAfOC+QOABe6+ImKxCe6eVs5mX3L3myu7MxK7/pWxm/94cQUXdGvJE+MvoH68Oq0isSya/6HJQIa7b3T3o8CLwOhy2s0AHgbyTrCeccD8U6pSarT0bfuY/lwaiW2aMHvSUBo3iA+7JBGpQDTh0BnIjBjPCqaVMLPBQFd3X3iS9VxL2XB4Jjh1dLcdf45hjJl9amavmlnX8lZmZjeaWZqZpWVnZ0exGxKGzbsPMfmZpbRo0oDnpg6jeZP6YZckIlGIJhzKOzHsJTPN4oBHgdtPuAKzYUCuu6dHTJ7g7gOAi4LHxGD6G0Ciu58PvAvMKW+d7v6Uuye5e1Lbtm2j2A2pbrv25zFx9hIKHZ6blkyH5o3CLklEohRNOGQBke/euwDbI8YTgP7AIjPbDAwHUoovSgfGUqrX4O7bgr8HgBcoOn2Fu+9x9yNBs5nAkGh3RmLHvsPHuH72UvYcPMozk4dyTtuzwi5JRCohmnBIBXqZWQ8za0DRC31K8Ux33+fubdw90d0TgcXAqOILzUHP4hqKrlUQTKtnZm2C4frAd4H0YLxjxLZHAWtPY/8kBHnHCpj+XBobsg/yp4lDGNi1RdgliUglVfhpJXfPN7ObgbeBeGC2u682s/uANHdPOfkaGAFkufvGiGkNgbeDYIin6PTRzGDerWY2CsgHcoDJldkhCVd+QSG3zF9O6uYcfj92MBf10ik/kZrI3L3iVjEuKSnJ09LK+0SsVCd3584/r+KltEzuvbIvk7/aI+ySROQkzGyZuyeVN08fNpcq88jf1/FSWia3XNJTwSBSwykcpErM/ucmnvjHBsYld+O2y3qHXY6InCaFg5y215dv476FaxjZrwP3X9Vft8UQqQUUDnJaFq3bxR2vrGT42a343dhBxMcpGERqA4WDnLLlW7/kR89/Qu/2CTx1fRKN6uu2GCK1hcJBTknGrgNMfTaVtgkNeXbqUJo10m0xRGoThYNU2va9h7l+1lLi4+KYOy2Zdgm6LYZIbaNwkEr58tBRrp+9lAN5+Tw7ZSjdWzcNuyQROQMq/Ia0SLHco/lMnZPK1pxc5kxJpn/n5mGXJCJniHoOEpXi+yWtzNzLY2MHceE5rcMuSUTOIPUcpEJH8gv40fPL+NeGPTxy9UBG9u9Y8UIiUqOp5yAndaygkFteWM4/1mXzwFUDGDOkS9gliUg1UDjICeUXFPIfL63g72t2cu+VfRk/rFvYJYlINVE4SLkKC52fvvopf/10Bz/7Th/dSE+kjlE4SBmFhc7PXlvFX5Zv4/bLenPjiHPCLklEqpnCQY7j7vzyjdW8mJrJzd/oyS2X9gq7JBEJgcJBSrg7v3pzLXM+3sL0i3pw+7d0622RukrhICV++856Zn64iUkXdudn3zlPt94WqcMUDgLAH977nD+8n8G45K784sp+CgaROk7hIDz1vxv4n3fW8/3BnXngqgHE6TcZROo8hUMd9+xHm/jVm5/x3fM78vDV5ysYRASIMhzMbKSZrTOzDDO78yTtrjYzN7OkYHyCma2IeBSa2aBg3qJgncXz2gXTG5rZS8G2lphZ4unvppTnhSVbufeNNXyrb3sevXYQ9eL1XkFEilT4amBm8cATwOVAX2CcmfUtp10CcCuwpHiau89z90HuPgiYCGx29xURi00onu/uu4Jp04Av3b0n8Cjw0Cnum5zEq8uyuOv1VXzj3Lb8Yfxg6isYRCRCNK8IyUCGu29096PAi8DoctrNAB4G8k6wnnHA/Ci2NxqYEwy/ClxqujpapVJWbuenr67kq+e04cnrhtCwnn7eU0SOF004dAYyI8azgmklzGww0NXdF55kPddSNhyeCU4p3R0RACXbc/d8YB9Q5v7QZnajmaWZWVp2dnYUuyEAb6V/wU9eWkFSYiueun6IfvdZRMoVTTiU967dS2aaxVF0+uf2E67AbBiQ6+7pEZMnuPsA4KLgMTGa7ZVMcH/K3ZPcPalt27YV74Xw/mc7uWX+J5zfpTmzJw+lSQPdsV1EyhdNOGQBXSPGuwDbI8YTgP7AIjPbDAwHUoovSgfGUqrX4O7bgr8HgBcoOn113PbMrB7QHMiJbnfkRD78PJsfPv8JfTo049kpyZzVUMEgIicWTTikAr3MrIeZNaDohT6leKa773P3Nu6e6O6JwGJglLunQUnP4hqKrlUQTKtnZm2C4frAd4HiXkUKMCkYvhp4393L9Bwkeos37mH6c2mc3aYpc6cl07xx/bBLEpEYV+HbR3fPN7ObgbeBeGC2u682s/uANHdPOfkaGAFkufvGiGkNgbeDYIgH3gVmBvNmAXPNLIOiHsPYSu2RHGfZlhymPptK15ZNmHfDMFo0aRB2SSJSA1hteFOelJTkaWlpYZcRcz7N2suEmUtok9CQl24cTrtmjcIuSURiiJktc/ek8ubpw+211Jrt+5k4ayktmtbnhenDFAwiUikKh1ro850HuG7WEpo2iOeFG4bTsXnjsEsSkRpG4VDLbMw+yPinl1Avzpg3fThdWzUJuyQRqYEUDrXI1j25jJ+5hMJC54Xpw+jRpmnYJYlIDaUPu9cS2/YeZtzMxeTlFzB/+nB6tksIuyQRqcHUc6gFdu7PY/zMxezPO8bcqcM4r2OzsEsSkRpO4VDDZR84wviZi9l94AhzpiYzoEvzsEsSkVpAp5VqsJxDR7nu6SVs35vHnKnJXNCtZdgliUgtoZ5DDbXv8DEmzlrC5j2HeHpSEsk9WoVdkojUIgqHGuhA3jEmzV7K5zsP8qeJQ/hqzzZhlyQitYzCoYbJPZrP1GdTSd+2j8fHD+bic9uFXZKI1EIKhxok71gBN8xJY9mWL/n92MF8q1+HsEsSkVpKF6RriCP5Bdw0dxkfb9zDb38wkCvO7xh2SSJSi6nnUAMcKyjkx/OW88H6bB78/gC+N7hL2CWJSC2ncIhx+QWF/PuLy3l37U5mjO7HtUO7hV2SiNQBCocYVlDo3P7KSt5c9QU/v+I8Jl6YGHZJIlJHKBxiVGGh899/+ZQFK7bzn98+lxsuOjvskkSkDlE4xCB3556UdF5Oy+LWS3vx42/0DLskEaljFA4xxt2ZsXAtzy/eyk1fP5uffLNX2CWJSB2kcIgh7s7Db69j9kebmPLVRO4c2QczC7ssEamDogoHMxtpZuvMLMPM7jxJu6vNzM0sKRifYGYrIh6FZjao1DIpZpYeMX6vmW2LWOY7p7pzNc1j72Xw5KINTBjWjXu+21fBICKhqfBLcGYWDzwBXAZkAalmluLua0q1SwBuBZYUT3P3ecC8YP4AYIG7r4hY5vvAwXI2+6i7P1L53am5nly0gUffXc/VQ7owY3R/BYOIhCqankMykOHuG939KPAiMLqcdjOAh4G8E6xnHDC/eMTMzgJuA+6vVMW10Ox/buKhtz5j1MBOPDTmfOLiFAwiEq5owqEzkBkxnhVMK2Fmg4Gu7r7wJOu5lohwoChM/gfILaftzWb2qZnNNrNa/SMFzy/ewn0L13B5/w789gcDiVcwiEgMiCYcynu18pKZZnHAo8DtJ1yB2TAg193Tg/FBQE93f62c5k8C5wCDgB0UBUh567zRzNLMLC07OzuK3Yg9L6dl8vPX07m0Tzt+P3Yw9eL1+QARiQ3RvBplAV0jxrsA2yPGE4D+wCIz2wwMB1KKL0oHxnJ8r+FCYEjQ/p9AbzNbBODuO929wN0LgZkUndYqwwlkV1oAAAqgSURBVN2fcvckd09q27ZtFLsRWxas2MZ//flTLurVhicmXECDegoGEYkd0bwipQK9zKyHmTWg6IU+pXimu+9z9zbunujuicBiYJS7p0FJz+Iaiq5VFC/zpLt3Ctp/DVjv7hcH7SNvN/o9IJ1a5m+rdnDbyysZ1qMVT01MolH9+LBLEhE5ToWfVnL3fDO7GXgbiAdmu/tqM7sPSHP3lJOvgRFAlrtvjLKmh4PTTg5sBm6Kcrka4d01O7ll/nIGdW3BrElDadxAwSAiscfcveJWMS4pKcnT0tLCLqNCH6zPZvqcNM7rmMDcG4bRrFH9sEsSkTrMzJa5e1J583Siu5r8a8NubnwujZ7tzuK5qQoGEYltCodqkLo5h2nPptG9dROev2EYzZsoGEQktikczrAVmXuZ8kwqHVs0Yt4Nw2nVtEHYJYmIVEjhcAalb9vH9bOW0KppA164YThtExqGXZKISFQUDmfIui8OMHHWEhIa1eeF6cPo0LxR2CWJiERN4XAGZOw6yISnF9OgXhwvTB9Gl5ZNwi5JRKRSFA5VbMueQ0x4ejFgvDB9ON1bNw27JBGRSlM4VKGsL3MZP3MJR/MLmXfDMM5pe1bYJYmInBKFQxX5Yl8e42cu4UDeMeZOG8a5HRLCLklE5JRVePsMqdiuA3mMn7mYnENHef6GYfTv3DzskkRETot6Dqdpz8EjXPf0Er7Yn8ezU4YyqGuLsEsSETltCofTsDf3KBNnLWXLnlxmTRpKUmKrsEsSEakSCodTtD/vGJNmLyVj10FmXp/Ehee0DrskEZEqo3A4BYeO5DPlmVRWb9/Pk9ddwIjeNe/HhkRETkYXpCvp8NECps1JZUXmXh4fN5hLz2sfdkkiIlVOPYdKyDtWwI1z01iyKYff/mAglw/oWPFCIiI1kMIhSkfzC/nxvE/48PPdPDzmfEYP6hx2SSIiZ4zCIQr5BYXcOn857322iwe+159rkrqGXZKIyBmlcKhAQaFz28sreWv1F9zz3b5MGNY97JJERM44hcNJFBY6//XnT0lZuZ07L+/D1K/1CLskEZFqoXA4AXfn5wvSeXVZFj/5Zm9++PVzwi5JRKTaRBUOZjbSzNaZWYaZ3XmSdlebmZtZUjA+wcxWRDwKzWxQqWVSzCw9YryVmb1jZp8Hf1ue6s6dKnfnvoVreGHJVv7t4nO49dKe1V2CiEioKgwHM4sHngAuB/oC48ysbzntEoBbgSXF09x9nrsPcvdBwERgs7uviFjm+8DBUqu6E3jP3XsB7wXj1cbdefCtz3jmo83c8LUe/Oe3z8XMqrMEEZHQRdNzSAYy3H2jux8FXgRGl9NuBvAwkHeC9YwD5hePmNlZwG3A/aXajQbmBMNzgKuiqLHKPPru5/zpg41MHN6du644T8EgInVSNOHQGciMGM8KppUws8FAV3dfeJL1XEtEOFAUJv8D5JZq197ddwAEf9uVtzIzu9HM0swsLTs7O4rdqNgT/8jgsfc+59qkrvxyVD8Fg4jUWdGEQ3mvkF4y0ywOeBS4/YQrMBsG5Lp7ejA+COjp7q9VrtyIAtyfcvckd09q2/b072309Icb+c3b6/je4M786vsDiItTMIhI3RVNOGQBkd/66gJsjxhPAPoDi8xsMzAcSCm+KB0Yy/G9hguBIUH7fwK9zWxRMG+nmXUECP7uinZnTtVzH2/m/r+u5YoBHfnN1ecTr2AQkToumnBIBXqZWQ8za0DRC31K8Ux33+fubdw90d0TgcXAKHdPg5KexTUUXasoXuZJd+8UtP8asN7dLw5mpwCTguFJwILT2L8KvZS6lXsWrOayvu353dhB1IvXp3tFRCp8JXT3fOBm4G1gLfCyu682s/vMbFQU2xgBZLn7xihrehC4zMw+By4Lxs+I15dv486/rOLrvdvy+PjB1FcwiIgAYO5ecasYl5SU5GlpaZVebummHJ7+cCOPjRtMo/rxZ6AyEZHYZWbL3D2pvHl1+vccknu0IrmHftpTRKQ0nUcREZEyFA4iIlKGwkFERMpQOIiISBkKBxERKUPhICIiZSgcRESkDIWDiIiUUSu+IW1m2cCWU1y8DbC7CsupKqqrclRX5cVqbaqrck6nru7uXu5trWtFOJwOM0s70dfHw6S6Kkd1VV6s1qa6KudM1aXTSiIiUobCQUREylA4wFNhF3ACqqtyVFflxWptqqtyzkhddf6ag4iIlKWeg4iIlKFwEBGRMmp1OJjZSDNbZ2YZZnZnOfMnm1m2ma0IHjdEzJtkZp8Hj0mllw2xroKI6Smllz2TdQVtfmBma8xstZm9EDE9tONVQV2hHS8zezRi2+vNbG/EvDCfXyerK8zj1c3M/mFmy83sUzP7TsS8/w6WW2dm346Fusws0cwORxyvP1ZzXd3N7L2gpkVm1iVi3uk/v9y9Vj6AeGADcDbQAFgJ9C3VZjLweDnLtgI2Bn9bBsMtw64rmHcwxOPVC1hefCyAdjFyvMqtK+zjVar9LcDsWDheJ6or7ONF0YXVHwXDfYHNEcMrgYZAj2A98TFQVyKQHuLxegWYFAxfAsytyudXbe45JAMZ7r7R3Y8CLwKjo1z228A77p7j7l8C7wAjY6CuMymauqYDTwTHBHffFUwP+3idqK4zqbL/juOA+cFw2MfrRHWdSdHU5UCzYLg5sD0YHg286O5H3H0TkBGsL+y6zqRo6uoLvBcM/yNifpU8v2pzOHQGMiPGs4JppY0JumWvmlnXSi5b3XUBNDKzNDNbbGZXVVFN0dbVG+htZh8F2x9ZiWXDqAvCPV5AUfefone871d22WquC8I9XvcC15lZFvAmRb2aaJcNoy6AHsHppg/M7KIqqinaulYCY4Lh7wEJZtY6ymUrVJvDwcqZVvpzu28Aie5+PvAuMKcSy4ZRF0A3L/qq/Hjgd2Z2TjXWVY+iUzgXU/SO82kzaxHlsmHUBeEer2JjgVfdveAUlq2s06kLwj1e44Bn3b0L8B1grpnFRblsGHXtoOh4DQZuA14ws2ZUjWjqugP4upktB74ObAPyo1y2QrU5HLKAyHfcXSjVHXT3Pe5+JBidCQyJdtmQ6sLdtwd/NwKLgMHVVVfQZoG7Hwu69+soelEO9XidpK6wj1exsRx/6ibs43WiusI+XtOAl4Ptfww0ouimcmEfr3LrCk5z7QmmL6PoGkHv6qrL3be7+/eDcLormLYvyn2q2Jm4mBILD4reTW6kqNtcfEGnX6k2HSOGvwcs9v+7oLOJoos5LYPhVjFQV0ugYTDcBvick1xsPAN1jQTmRGw/E2gdA8frRHWFeryCducCmwm+cBoLz6+T1BX28+tvwORg+DyKXtAM6MfxF6Q3UnUXpE+nrrbFdVB04XhbNT/v2wBxwfADwH1V+fw67Z2I5QdFXcD1FCX6XcG0+4BRwfCvgdXBgf8H0Cdi2akUXfjKAKbEQl3AV4BVwfRVwLRqrsuA3wJrgu2PjZHjVW5dYR+vYPxe4MFylg3teJ2orrCPF0UXWD8Ktr8C+FbEsncFy60DLo+Fuig631/8//QT4MpqrutqigJ8PfA0QbBX1fNLt88QEZEyavM1BxEROUUKBxERKUPhICIiZSgcRESkDIWDiIiUoXAQEZEyFA4iIlLG/weGRp1DPCbjcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "#train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "#train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再调小一点，由于特征包括原始特征+tfidf特征，是多了些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   3 out of   6 | elapsed:  2.1min remaining:  2.1min\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   6 | elapsed:  3.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=1.0,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=30,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.8,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4, param_grid={'colsample_bytree': [0.3, 0.4]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(3,5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4729705008983489\n",
      "{'colsample_bytree': 0.3}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 3648\n",
      "best cv score: 0.4645050558729036\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有训练数据，采用最佳参数重新训练模型\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "               colsample_bytree=0.4, importance_type='split',\n",
       "               learning_rate=0.01, max_bin=127, max_depth=7,\n",
       "               min_child_samples=40, min_child_weight=0.001, min_split_gain=0.0,\n",
       "               n_estimators=3648, n_jobs=4, num_class=9, num_leaves=75,\n",
       "               objective='multiclass', random_state=None, reg_alpha=0.0,\n",
       "               reg_lambda=0.0, silent=False, subsample=0.8,\n",
       "               subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':75,\n",
    "          'min_child_samples':40,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle.dump(lg, open(\"Otto_LightGBM_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"columns\":list(feat_names), \"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>159</td>\n",
       "      <td>feat_67_tfidf</td>\n",
       "      <td>37470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>117</td>\n",
       "      <td>feat_25_tfidf</td>\n",
       "      <td>36468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>116</td>\n",
       "      <td>feat_24_tfidf</td>\n",
       "      <td>33995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>140</td>\n",
       "      <td>feat_48_tfidf</td>\n",
       "      <td>33585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>132</td>\n",
       "      <td>feat_40_tfidf</td>\n",
       "      <td>31993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>feat_61</td>\n",
       "      <td>695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>81</td>\n",
       "      <td>feat_82</td>\n",
       "      <td>594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>feat_6</td>\n",
       "      <td>441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>83</td>\n",
       "      <td>feat_84</td>\n",
       "      <td>439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>feat_51</td>\n",
       "      <td>277</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           columns  importance\n",
       "159  feat_67_tfidf       37470\n",
       "117  feat_25_tfidf       36468\n",
       "116  feat_24_tfidf       33995\n",
       "140  feat_48_tfidf       33585\n",
       "132  feat_40_tfidf       31993\n",
       "..             ...         ...\n",
       "60         feat_61         695\n",
       "81         feat_82         594\n",
       "5           feat_6         441\n",
       "83         feat_84         439\n",
       "50         feat_51         277\n",
       "\n",
       "[186 rows x 2 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXj0lEQVR4nO3df4zcd53f8efrnB9HDzg7ZEGW7Z4Dda8EpHOCm1iih2iAxAntObRclagi1jWV7zhHAvVaYYrUcECk0AqQIkFOoXFxThST8kOxLqbGyuWKkMiPDTg/TEi9mFxj4sYGhxBEG5rw7h/zWZg4s7uz692dmd3nQxrNzHu+39n3fGfm+5rv5/ud2VQVkqTl7TcG3YAkafAMA0mSYSBJMgwkSRgGkiTgjEE3MFfnnnturV+/ftBtSNJIeeCBB35UVWOn1kc2DNavX8/4+Pig25CkkZLkb3vVHSaSJBkGkiTDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgaRlbv3OO1m/885BtzFwhoEkyTCQJBkGkiQMA0kShoG07LizVL0YBpIkw0CSZBhIkjAMJEn0EQZJfjPJfUkeTHIoyZ+3+ueS/CDJwXba2OpJclOSiSQPJbmw6762JTncTtu66m9K8nCb56YkWYgHK0nq7Yw+pnkOuKSqfpbkTOCbSb7Wbvt3VfWlU6a/HNjQThcDNwMXJzkHuB7YBBTwQJK9VfV0m2Y7cA+wD9gCfA1J0qKYccugOn7Wrp7ZTjXNLFuB29p89wArk6wGLgMOVNXJFgAHgC3ttldW1beqqoDbgCtP4zFJkmapr30GSVYkOQgcp7NCv7fddEMbCvpUkrNbbQ3wRNfsR1ttuvrRHvVefWxPMp5k/MSJE/20LknqQ19hUFUvVNVGYC1wUZI3Ah8E/gHwD4FzgA+0yXuN99cc6r36uKWqNlXVprGxsX5alyT1YVZHE1XVT4C/AbZU1bE2FPQc8F+Ai9pkR4F1XbOtBZ6cob62R12StEj6OZpoLMnKdvllwNuB77WxftqRP1cCj7RZ9gLXtKOKNgPPVNUxYD9waZJVSVYBlwL7223PJtnc7usa4I75fZiSpOn0czTRamB3khV0wuP2qvqrJH+dZIzOMM9B4E/a9PuAK4AJ4OfAHwFU1ckkHwXub9N9pKpOtsvvBT4HvIzOUUQeSSRJi2jGMKiqh4ALetQvmWL6AnZMcdsuYFeP+jjwxpl6kSQtDL+BLC0T/lqppmMYSJIMA0mD5z+lHzzDQJJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCTNM788NpoMA0mSYSBJMgwkSRgGkiQMA0kShoEkCcNAkkQfYZDkN5Pcl+TBJIeS/Hmrn5fk3iSHk3wxyVmtfna7PtFuX991Xx9s9ceSXNZV39JqE0l2zv/DlCRNp58tg+eAS6rq94CNwJYkm4GPA5+qqg3A08C1bfprgaer6u8Bn2rTkeR84CrgDcAW4DNJViRZAXwauBw4H7i6TStJWiQzhkF1/KxdPbOdCrgE+FKr7waubJe3tuu029+WJK2+p6qeq6ofABPARe00UVVHquoXwJ42rSRpkfS1z6B9gj8IHAcOAN8HflJVz7dJjgJr2uU1wBMA7fZngFd110+ZZ6p6rz62JxlPMn7ixIl+WpekoTYsP9/RVxhU1QtVtRFYS+eT/Ot7TdbOM8Vts6336uOWqtpUVZvGxsZmblyS1JdZHU1UVT8B/gbYDKxMcka7aS3wZLt8FFgH0G7/beBkd/2UeaaqS5IWST9HE40lWdkuvwx4O/AocDfw7jbZNuCOdnlvu067/a+rqlr9qna00XnABuA+4H5gQzs66Sw6O5n3zseDkyT154yZJ2E1sLsd9fMbwO1V9VdJvgvsSfIx4DvArW36W4G/TDJBZ4vgKoCqOpTkduC7wPPAjqp6ASDJdcB+YAWwq6oOzdsjlCTNaMYwqKqHgAt61I/Q2X9wav3/An84xX3dANzQo74P2NdHv5KkBeA3kKURMSxHnWhpMgwkSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJJEH2GQZF2Su5M8muRQkve1+oeT/DDJwXa6omueDyaZSPJYksu66ltabSLJzq76eUnuTXI4yReTnDXfD1SSNLV+tgyeB/6sql4PbAZ2JDm/3fapqtrYTvsA2m1XAW8AtgCfSbIiyQrg08DlwPnA1V338/F2XxuAp4Fr5+nxSZL6MGMYVNWxqvp2u/ws8CiwZppZtgJ7quq5qvoBMAFc1E4TVXWkqn4B7AG2JglwCfClNv9u4Mq5PiBJ0uzNap9BkvXABcC9rXRdkoeS7EqyqtXWAE90zXa01aaqvwr4SVU9f0pdkrRI+g6DJC8Hvgy8v6p+CtwMvA7YCBwDPjE5aY/Zaw71Xj1sTzKeZPzEiRP9ti5JmkFfYZDkTDpB8Pmq+gpAVT1VVS9U1S+Bz9IZBoLOJ/t1XbOvBZ6cpv4jYGWSM06pv0RV3VJVm6pq09jYWD+tS5L60M/RRAFuBR6tqk921Vd3TfYu4JF2eS9wVZKzk5wHbADuA+4HNrQjh86is5N5b1UVcDfw7jb/NuCO03tYkqTZOGPmSXgz8B7g4SQHW+3f0zkaaCOdIZ3HgT8GqKpDSW4HvkvnSKQdVfUCQJLrgP3ACmBXVR1q9/cBYE+SjwHfoRM+kqRFMmMYVNU36T2uv2+aeW4AbuhR39drvqo6wq+HmSRJi8xvIEuSDANJkmEgScIwkCRhGEha4tbvvHPQLYwEw0CSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiT6++c2koZE908rPH7jOwfYiZYatwwkSYaBhp8/NCYtPMNAkjRzGCRZl+TuJI8mOZTkfa1+TpIDSQ6381WtniQ3JZlI8lCSC7vua1ub/nCSbV31NyV5uM1zU5Je/3NZ0oC4dbb09bNl8DzwZ1X1emAzsCPJ+cBO4K6q2gDc1a4DXA5saKftwM3QCQ/geuBi4CLg+skAadNs75pvy+k/NEl6sfU77zTYpjBjGFTVsar6drv8LPAosAbYCuxuk+0GrmyXtwK3Vcc9wMokq4HLgANVdbKqngYOAFvaba+sqm9VVQG3dd2XJGkRzGqfQZL1wAXAvcBrquoYdAIDeHWbbA3wRNdsR1ttuvrRHvVef397kvEk4ydOnJhN65KkafQdBkleDnwZeH9V/XS6SXvUag71lxarbqmqTVW1aWxsbKaWJUl96isMkpxJJwg+X1VfaeWn2hAP7fx4qx8F1nXNvhZ4cob62h51SSPOMfrR0c/RRAFuBR6tqk923bQXmDwiaBtwR1f9mnZU0WbgmTaMtB+4NMmqtuP4UmB/u+3ZJJvb37qm674kSYugny2DNwPvAS5JcrCdrgBuBN6R5DDwjnYdYB9wBJgAPgv8KUBVnQQ+CtzfTh9pNYD3Av+5zfN94Gvz8NgkzSM/5S+8QS7fGX+bqKq+Se9xfYC39Zi+gB1T3NcuYFeP+jjwxpl6kSQtDL+BLEkyDCRJhoEkCcNAkoRhIEnCMJA0Qjy0deEYBpIkw0CSZBhI0oyWw/CUYSBJMgyGyXL49CFpOBkGkiTDQJJkGEgD5/CghoFhIEkyDNQfP71qlPh6nT3DQJJkGEiamp+wlw/DQJI0cxgk2ZXkeJJHumofTvLDJAfb6Yqu2z6YZCLJY0ku66pvabWJJDu76ucluTfJ4SRfTHLWfD7AUeM/HZc0CP1sGXwO2NKj/qmq2thO+wCSnA9cBbyhzfOZJCuSrAA+DVwOnA9c3aYF+Hi7rw3A08C1p/OAJEmzN2MYVNU3gJN93t9WYE9VPVdVPwAmgIvaaaKqjlTVL4A9wNYkAS4BvtTm3w1cOcvHIEnzajluoZ/OPoPrkjzUhpFWtdoa4ImuaY622lT1VwE/qarnT6n3lGR7kvEk4ydOnDiN1iVJ3eYaBjcDrwM2AseAT7R6ekxbc6j3VFW3VNWmqto0NjY2u44lSVOaUxhU1VNV9UJV/RL4LJ1hIOh8sl/XNela4Mlp6j8CViY545S6FtBy2/zVwvB1tLTMKQySrO66+i5g8kijvcBVSc5Och6wAbgPuB/Y0I4cOovOTua9VVXA3cC72/zbgDvm0pMkae7OmGmCJF8A3gqcm+QocD3w1iQb6QzpPA78MUBVHUpyO/Bd4HlgR1W90O7nOmA/sALYVVWH2p/4ALAnyceA7wC3ztujkyT1ZcYwqKqre5SnXGFX1Q3ADT3q+4B9PepH+PUw08ia3GR+/MZ3DrgTSZo9v4EsTcNxcfWyFA89NQwkLbiltuJcigwDaQgsxU+aGi2GgSTJMJCkqQzL1tpibDkaBpIkw0CSBmFYtjomGQbSEjZsKxwNL8NAkmQYSKPOT/+aD4aBNIQWegXv9xp0qhl/m0iSBs3gWniGgTSiXEGePpfhrzlMJEkyDDR4fjrTMFjur0PDQJJkGEiSDANJGrhhGKKaMQyS7EpyPMkjXbVzkhxIcridr2r1JLkpyUSSh5Jc2DXPtjb94STbuupvSvJwm+emJJnvBylJC2EYVuLzpZ8tg88BW06p7QTuqqoNwF3tOsDlwIZ22g7cDJ3wAK4HLqbz/46vnwyQNs32rvlO/VuSpAU2YxhU1TeAk6eUtwK72+XdwJVd9duq4x5gZZLVwGXAgao6WVVPAweALe22V1bVt6qqgNu67mskLaVPCoPg8hstfpN56ZjrPoPXVNUxgHb+6lZfAzzRNd3RVpuufrRHXdIImQyE7mAwJEbLfO9A7jXeX3Oo977zZHuS8STjJ06cmGOLS5dvPg07X6PDa65h8FQb4qGdH2/1o8C6runWAk/OUF/bo95TVd1SVZuqatPY2NgcW5cknWquYbAXmDwiaBtwR1f9mnZU0WbgmTaMtB+4NMmqtuP4UmB/u+3ZJJvbUUTXdN3XyHIcVRoM33tzN+MP1SX5AvBW4NwkR+kcFXQjcHuSa4H/Bfxhm3wfcAUwAfwc+COAqjqZ5KPA/W26j1TV5E7p99I5YullwNfaSZK0iGYMg6q6eoqb3tZj2gJ2THE/u4BdPerjwBtn6kO9rd95J4/f+M5Bt6Euk59MB/G8DPJva7T5DWQNhJvymi++luaHYSBp6LiCX3yGgaRZcUW9NBkGUp8GdaSKK9+XGtTzsJSfC8OgT0v1RbBUH5ek2TEMNK8MF2k0GQaStEj6+bA0qOEow0CSZmGpbv3O+KUzDb+l+uKUpuPrfn65ZSAtAldcGnaGgYDRXlkt9UP+ljqfu+FgGMzAFc1o8jmbPZdZx3wth1FbnoaBRtKovdGk2Vrs17hhIElDbLFCwTCQtOy4ZflShsES5b6O0eRzNrpG/T1nGEjSPBrVQDAMJL3IqK7MRrXvYWEYSAvMlZRGwWmFQZLHkzyc5GCS8VY7J8mBJIfb+apWT5KbkkwkeSjJhV33s61NfzjJttN7SJJGlcE5OPOxZfCPq2pjVW1q13cCd1XVBuCudh3gcmBDO20HboZOeADXAxcDFwHXTwbIsBr1HUXqz3J4jgf5GJfD8h0lCzFMtBXY3S7vBq7sqt9WHfcAK5OsBi4DDlTVyap6GjgAbFmAvkaKb5TR4vOlUXe6YVDA15M8kGR7q72mqo4BtPNXt/oa4ImueY+22lT1l0iyPcl4kvETJ06cZuvScDudLVDDSbN1umHw5qq6kM4Q0I4kb5lm2vSo1TT1lxarbqmqTVW1aWxsbPbdNr5RRkM/K8PTvV1Sx2mFQVU92c6PA1+lM+b/VBv+oZ0fb5MfBdZ1zb4WeHKauvQirtilhTPnMEjyW0leMXkZuBR4BNgLTB4RtA24o13eC1zTjiraDDzThpH2A5cmWdV2HF/aakveUtoRvVQex3K02K/D2f49X1uL43S2DF4DfDPJg8B9wJ1V9d+BG4F3JDkMvKNdB9gHHAEmgM8CfwpQVSeBjwL3t9NHWk0jzjfx9Fw+i8vlPb05/9vLqjoC/F6P+o+Bt/WoF7BjivvaBeyaay+DNtWLbP3OO3n8xnf+6nyQJnscdB9afK4EF95SWMZ+A3kJWgovzKVqIZ8bn3edDsNA6oMrWi11hoEW1emuVBdyZ+dS2qGv4TDfr6mFfH0aBkvcdPszXPFJmmQY6Ff6DYjZhshih85i/T2/8KalxDBYZK4gBs/nQHqpOR9aquVhLivOpb6y7T5UeKk+1qX6uDQ1twy0ZLgCk+bOMGj6XZG4wpmbYdhhPei/r+VrFF57hsGQG4UX0SgahnCSholhIElyB/KofTpcLr8xNGrPi4aTr6P+LfswGBSP0lm+fB6Xt2F9/h0mWiSjNEY92WevfkflMUiaHbcMptHPN0znY7hmGH7ieq7mKxz8NU9psJb1loErCc03X1Pzx2W5uJZ1GJxqWIdyHK6RtNAMAy1rhqrU4T4DDYwrYml4DM2WQZItSR5LMpFk5yB7cSW1/HQ/5z7/Wo6GIgySrAA+DVwOnA9cneT8wXalmbjSlJaOoQgD4CJgoqqOVNUvgD3A1gH3JEnLRqpq0D2Q5N3Alqr61+36e4CLq+q6U6bbDmxvV38XeOw0/uy5wI9OY/7FMOw9Dnt/YI/zxR7nxzD0+DtVNXZqcVh2IKdH7SUpVVW3ALfMyx9Mxqtq03zc10IZ9h6HvT+wx/lij/NjmHsclmGio8C6rutrgScH1IskLTvDEgb3AxuSnJfkLOAqYO+Ae5KkZWMohomq6vkk1wH7gRXArqo6tMB/dl6GmxbYsPc47P2BPc4Xe5wfQ9vjUOxAliQN1rAME0mSBsgwkCQtvzAYpp+9mJRkXZK7kzya5FCS97X6h5P8MMnBdrpiwH0+nuTh1st4q52T5ECSw+181QD7+92uZXUwyU+TvH/QyzHJriTHkzzSVeu53NJxU3t9PpTkwgH2+J+SfK/18dUkK1t9fZL/07U8/2KAPU753Cb5YFuOjyW5bIA9frGrv8eTHGz1gSzHKVXVsjnR2Tn9feC1wFnAg8D5Q9DXauDCdvkVwP+k87McHwb+7aD76+rzceDcU2r/EdjZLu8EPj7oPrue6/8N/M6glyPwFuBC4JGZlhtwBfA1Ot+92QzcO8AeLwXOaJc/3tXj+u7pBrwcez637f3zIHA2cF57368YRI+n3P4J4D8McjlOdVpuWwZD+bMXVXWsqr7dLj8LPAqsGWxXfdsK7G6XdwNXDrCXbm8Dvl9VfzvoRqrqG8DJU8pTLbetwG3VcQ+wMsnqQfRYVV+vqufb1XvofP9nYKZYjlPZCuypqueq6gfABJ33/4KarsckAf4F8IWF7mMullsYrAGe6Lp+lCFb6SZZD1wA3NtK17XN9F2DHIJpCvh6kgfaT4MAvKaqjkEn1IBXD6y7F7uKF7/phmk5wtTLbVhfo/+KzhbLpPOSfCfJ/0jy+4Nqqun13A7jcvx94KmqOtxVG5rluNzCoK+fvRiUJC8Hvgy8v6p+CtwMvA7YCByjs4k5SG+uqgvp/LrsjiRvGXA/PbUvLv4B8N9aadiW43SG7jWa5EPA88DnW+kY8Her6gLg3wD/NckrB9TeVM/t0C1H4Gpe/AFlmJbjsguDof3ZiyRn0gmCz1fVVwCq6qmqeqGqfgl8lkXYzJ1OVT3Zzo8DX239PDU5jNHOjw+uw1+5HPh2VT0Fw7ccm6mW21C9RpNsA/4J8C+rDXS3oZcft8sP0BmP//uD6G+a53bYluMZwD8DvjhZG6blCMsvDIbyZy/aWOKtwKNV9cmuevdY8buAR06dd7Ek+a0kr5i8TGfn4iN0lt+2Ntk24I7BdPgiL/oENkzLsctUy20vcE07qmgz8MzkcNJiS7IF+ADwB1X18676WDr/g4QkrwU2AEcG1ONUz+1e4KokZyc5j06P9y12f13eDnyvqo5OFoZpOQLL62ii9sHmCjpH63wf+NCg+2k9/SM6m7APAQfb6QrgL4GHW30vsHqAPb6WztEZDwKHJpcd8CrgLuBwOz9nwMvy7wA/Bn67qzbQ5UgnmI4B/4/OJ9Zrp1pudIY3Pt1enw8DmwbY4wSdcffJ1+RftGn/eXsNPAh8G/inA+xxyucW+FBbjo8Blw+qx1b/HPAnp0w7kOU41cmfo5AkLbthIklSD4aBJMkwkCQZBpIkDANJEoaBJAnDQJIE/H+Kb2CKlHwH4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(lg.feature_importances_)), lg.feature_importances_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tfidf的特征重要性更高一些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
